<![CDATA[Personal Programming Notes]]> 2022-04-05T09:27:45-07:00 http://tdongsi.github.io/ Octopress <![CDATA[Closures in Loop]]> 2021-07-24T22:12:43-07:00 http://tdongsi.github.io/blog/2021/07/24/closures-in-loop In this post, we’ll look at a nasty gotcha with closures that can ensare even the most experienced programmers. This problem can happen to any programming language that has closures.

Recently, while working in Jenkinsfile, I got stuck with this piece of Groovy code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

def deployedApps = [ 'app1', 'app2', 'app3']
def jobs = [:]

// ERROR: what's wrong with this?
for (String app: deployedApps) {
    jobs[app] = {
        println("Deploy $app")
    }
}

// DEBUG: execute the closure to debug.
for (k in deployedApps) {
    jobs[k]()
}

In the above code, the jobs variable is a mapping from String to Groovy Closure objects. It is intended for parallel step to programmatically create a multi-fork stage in Jenkins.

1
2
3
4
5

// Constructing "jobs" variable as above.

stage('Deploy') {
  parallel jobs
}

The stage will look like this in BlueOcean interface:

As you can probably guess, the intention is to concurrently deploy/print mulitple distinct applications, colorfully named as app1 app2 app3, in a Jenkins stage “Deploy”. However, it does not work, as shown in the console log output below (NOTE: the deployment code has been replaced with println for simplicity).

Console log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

[Pipeline] stage
[Pipeline] { (Deploy)
[Pipeline] parallel
[Pipeline] { (Branch: app1)
[Pipeline] { (Branch: app2)
[Pipeline] { (Branch: app3)
[Pipeline] echo
Deploy app3
[Pipeline] }
[Pipeline] echo
Deploy app3
[Pipeline] }
[Pipeline] echo
Deploy app3
[Pipeline] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage

Although the keys (used for display names) are correct, the values, which are Closure objects for actual execution such as deployment or simple prints, are wrong. The bug is subtle and puzzling: only the last element in the application list, regardless of its size and content, will be deployed or printed out (app3 in this example). As we look further into it, we’ll see that this problem has nothing to do with Map or Groovy. It can happen to any language that has closures. For example: The same above problem can be simplified with list in Groovy:

1
2
3
4
5
6

// List version
def closures = []
for (int i = 0; i < 5; i++) {
    closures += { println i }
}
closures.each{ it() }

The same problem can be seen in Go language:

1
2
3
4
5
6
7
8
9
10
11
12
13

var closures []func()

// Functions to Print from 0 to 4
for i := 0; i < 5; i++ {
    closures = append(closures, func() {
        fmt.Println(i)
    })
}

// Now call them
for _, f := range closures {
    f()
}

or in JavaScript:

1
2
3
4
5

for (var i = 0; i < 5; i++) {
    setTimeout(function() {
        console.log(i);
    }, 0);
}

In these List-based examples, “5” (the last values of the list) is always printed 5 times.

It turns out that this surprise problem is quite common. In fact, it is so common that the “Go Programming Language” book dedicates a whole section (5.6.1: Caveat: Capturing iteration variables) in its Chapter 5 to discuss this gotcha. The reason is related to scope rule: as we iterate through closures and use iteration variable (i in the three list examples), all the Closure objects created in this loop “capture” and share the same variable i (i.e., same addressable memory location) - not its value at that particular iteration (such as 0 in the first iteration). At the end of the loop, the variable i has been updated several times and has the final value 5. Thus, the values that are used by all individual Closure objects when executed are all 5’s instead of 0-4 for each.

Now that we understand what went wrong, the fix is pretty simple: we simply declare a new variable within the loop body before using it in the closure. By doing so, each Closure object will have a separate variable (with distinct memory address) and value.

Groovy fix
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

// Map version
def deployedApps = [ 'app1', 'app2', 'app3']
def jobs = [:]

for (String e: deployedApps) {
    def app = e  // TRICKY: necessary!
    jobs[app] = {
        println("Deploy $app")
    }
}

// List version
def closures = []
for (int i = 0; i < 5; i++) {
    def e = i  // TRICKY: necessary!
    closures += { println e }
}
closures.each{ it() }
Go fix
1
2
3
4
5
6
7
8
9

var closures []func()

// Functions to Print from 0 to 4
for i := 0; i < 5; i++ {
    e := i  // TRICKY: necessary!
    closures = append(closures, func() {
        fmt.Println(e)
    })
}
JavaScript fix
1
2
3
4
5
6

for (var i = 0; i < 5; i++) {
    let e = i;  // TRICKY: necessary!
    setTimeout(function() {
        console.log(e);
    }, 0);
}

In general, I would recommend adding the comment TRICKY: necessary!. This would caution another team member, out of desire for premature optimization, from accidentally remove the apparently useless line and produce the subtly incorrect variants as seen above.

]]> <![CDATA[Apple News in 2020 Election]]> 2020-12-20T02:08:21-08:00 http://tdongsi.github.io/blog/2020/12/20/apple-news-in-2020-election Some positive feedback about Apple News during 2020 election.

The feedback is overall positive.

Some are outright hilarious.

Twitter users generally like the clear, digestable format on Apple News. Others also start appreciating human-curated contents on Apple News, especially in the context of “fake news” and deeply divided 2020 election.

After the election, it is reported that the President-elect Joe Biden “relies on Apple News to help him get headlines from other reputable media sources.” Politico also mentioned Biden’s media diet, noting he is a “devoted fan” of Apple News and has news app notifications on his iPhone on to keep up with stories throughout the day.

P.S.: I’m just a little proud of my work’s impact. This post is not meant to express my political views.

]]> <![CDATA[Minikube in Corporate VPN]]> 2018-12-31T12:40:08-08:00 http://tdongsi.github.io/blog/2018/12/31/minikube-in-corporate-vpn If you are connected to corporate VPN via Cisco’s AnyConnect client, you might have problem with starting Minikube.

1
2
3
4
5
6
7

tdongsi$ minikube start --disk-size=50g --kubernetes-version=v1.8.0
Starting local Kubernetes v1.8.0 cluster...
Starting VM...
Downloading Minikube ISO
 140.01 MB / 140.01 MB [============================================] 100.00% 0s

^C

The issue has been extensively discussed in this bug report. This pull request supposedly fixes the issue, in v0.19.0 release. However, I’m still occasionally seeing the issue. I have attempted different approaches but they have different degrees of convenience and success in different networks.

  1. Use OpenConnect for VPN access rather than Cisco’s AnyConnect client.
  2. Set port forwarding to forward port 8443 on 127.0.0.1 to port 8443 in the minikube VM.
  3. Use --host-only-cidr option in minikube start.

In this post, we will look into each approach in more details.

Using OpenConnect

OpenConnect is a CLI client alternative for Cisco’s AnyConnect VPN. Here’s how you setup OpenConnect on Mac OSX:

  1. OpenConnect can be installed via homebrew:

     brew update
 brew install openconnect
  2. Install the Mac OS X TUN/TAP driver

  3. Connect. The only thing you should be prompted for is your VPN password.

     sudo openconnect --user=<VPN username> <your vpn hostname>
  4. To disconnect, just Ctrl-C in the window where you started the VPN connection.

Port forwarding localhost:xxx -> minikube_IP:xxx

This approach is the more convenient and more reliable in my experience. All you need to do is to set up a list of port forwarding rules for minikube’s VirtualBox:

1
2
3
4

VBoxManage controlvm minikube natpf1 k8s-apiserver,tcp,127.0.0.1,8443,,8443
VBoxManage controlvm minikube natpf1 k8s-dashboard,tcp,127.0.0.1,30000,,30000
VBoxManage controlvm minikube natpf1 jenkins,tcp,127.0.0.1,30080,,30080
VBoxManage controlvm minikube natpf1 docker,tcp,127.0.0.1,2376,,2376

Then, you can set up a new Kubernetes context for working with VPN:

1
2

kubectl config set-cluster minikube-vpn --server=https://127.0.0.1:8443 --insecure-skip-tls-verify
kubectl config set-context minikube-vpn --cluster=minikube-vpn --user=minikube

When working on VPN, you can set kubectl to switch to the new context:

1

kubectl config use-context minikube-vpn

All Minikube URLs now must be accessed through localhost in browser. For example, the standard Kubernetes dashboard URL such as:

1
2

tdongsi$ minikube dashboard --url
http://192.168.99.100:30000

must now be accessed via localhost:30000. Similar applies to other services that are deployed to minikube, such as jenkins shown above.

In addition, the eval $(minikube docker-env) standard pattern to reuse minikube’s Docker deamon would not work anymore.

1
2
3
4
5
6
7
8
9
10
11
12

tdongsi$ minikube docker-env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/Users/tdongsi/.minikube/certs"
export DOCKER_API_VERSION="1.23"
# Run this command to configure your shell:
# eval $(minikube docker-env)

tdongsi$ echo $DOCKER_HOST
tcp://192.168.99.100:2376
tdongsi$ docker images
Cannot connect to the Docker daemon at tcp://192.168.99.100:2376. Is the docker daemon running?

Instead, you have to adjust DOCKER_HOST accordingly and use docker --tlsverify=false ....

1
2
3
4
5

tdongsi$ export DOCKER_HOST="tcp://127.0.0.1:2376"
tdongsi$ alias dockervpn="docker --tlsverify=false"

tdongsi$ dockervpn images
...

Finally, when not working on VPN, you can set kubectl to switch back to the old context:

1

kubectl config use-context minikube

Use --host-only-cidr option

This approach is the most simple but it also has less success than I hoped. The idea of this approach is that AnyConnect VPN client likely routes 192.168.96.0/19 through its tunnel. This may conflict with the default Minikube network of 192.168.99.0/24. Therefore, we use minikube start --host-only-cidr 10.254.254.1/24 to instruct minikube to use a different, unused arbitrary network. It is worth a try but it often does not work in my experience.

Reference

]]> <![CDATA[Github REST API Cookbook]]> 2018-08-06T21:37:20-07:00 http://tdongsi.github.io/blog/2018/08/06/github-rest-api The blog post shows some useful snippets for interacting with Github API. Jenkins pipelines regularly interacts with Github (public or Enterprise) API to perform some query/posting, for example, regarding the current pull request. For that reason, some of the following snippets are either in Groovy or curl commands embedded in Groovy-based Jenkinsfile code with some Jenkinsfile DSLs.

Working with Pull Requests

Extracting Pull Request details

Get PR body text
1
2
3
4
5
6
7
8
9
10
11
12
13

import groovy.json.JsonSlurper

def getPrBody(String githubUsername, String githubToken, String repo, String id) {
  String GITHUB_API = 'https://git.enterprise.com/api/v3/repos'

  String url = "${GITHUB_API}/${githubUsername}/${repo}/pulls/${id}"
  println "Querying ${url}"
  def text = url.toURL().getText(requestProperties: ['Authorization': "token ${githubToken}"])
  def json = new JsonSlurper().parseText(text)
  def bodyText = json.body

  return bodyText
}

The equivalent curl command is as follows, with JSON processing is done in jq:

Equivalent curl | jq command in Jenkisfile
1
2
3

sh """
curl -s -H "Authorization: token ${env.GITHUB_TOKEN}" ${GITHUB_API}/${org}/${repo}/pulls/${env.CHANGE_ID} | jq '.body' -r > pr_body.txt
"""

Posting comment on the Pull Request

Reference: Create a comment.

Equivalent curl in Jenkinsfile
1
2
3

sh """
curl -s -X POST -H "Authorization: token ${env.GITHUB_TOKEN}" --data '{"body": "${comment}"}' ${GITHUB_API}/${org}/${repo}/issues/${env.CHANGE_ID}/comments
"""

Merge Pull Request

Based on this article.

Merge pull request
1
2
3
4
5
6
7
8

stage ("Merge PR") {
    steps {
        withCredentials([usernamePassword(credentialsId: 'credential-value', usernameVariable: 'ACCESS_TOKEN_USERNAME', passwordVariable: 'ACCESS_TOKEN_PASSWORD',)]) {
            def GITHUB = 'https://github.ibm.com/api/v3/repos'
            sh "curl -X PUT -d '{\"commit_title\": \"Merge pull request\"}' ${GITHUB}/org-name/repo-name/pulls/${env.CHANGE_ID}/merge?access_token=${env.ACCESS_TOKEN_PASSWORD}"
        }
    }
}

The Jenkins-provided environment variable $CHANGE_ID, in the case of a pull request, is the pull request number.

Working with Branches

Getting email of branch maintainer

At the end of a Jenkins build for a feature branch (NOT develop/master), you may want to email some developer of its status, as opposed to blasting a whole distribution list. Note that in Git, there is no such metadata for branch creator, as discussed here. Instead, it makes more sense to notify the latest/active committer which is likely the owner of the branch.

Get email of branch maintainer.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

def getBranchCreator(String githubUsername, String githubToken, String repo, String branch) {
    String GITHUB_API = 'https://git.enterprise.com/api/v3/repos'

    String url = "${GITHUB_API}/${githubUsername}/${repo}/branches/${branch}"
    println "Querying ${url}"
    def text = url.toURL().getText(requestProperties: ['Authorization': "token ${githubToken}"])
    def json = new JsonSlurper().parseText(text)

    // Get last committer.
    def creator = json.commit.commit.committer.email
    // TRICKY: json.commit.commit.committer.email is not valid if someone commits from Github web interface.
    // In the case, committer name is 'GitHub Enterprise'.
    if (json.commit.commit.committer.name == 'GitHub Enterprise') {
    // Use author's email instead
    creator = json.commit.commit.author.email
    }
    // TRICKY: the following can return inconsistent data, including "null".
    // def author = json.author
    return creator
}

// Calling from Jenkinsfile
withCredentials([
    [$class: 'UsernamePasswordMultiBinding', credentialsId: 'my-credentials',
        passwordVariable: 'GITHUB_PASSWORD', usernameVariable: 'GITHUB_USERNAME']
]) {
    if (env.BRANCH_NAME ==~ /PR-\d+/ ) {
        // If it is a PR build, use some distribution list
        email = 'someemail@enterprise.com'
    } else {
        // NOTE: Replace env.GITHUB_USERNAME with the correct Github org name.
        email = getBranchCreator(env.GITHUB_USERNAME, env.GITHUB_PASSWORD, 'my_repo', env.BRANCH_NAME)
    }
}

Deleting a branch

Searching how to delete a branch in Github API’s Branches reference does not return anything. In fact, to delete a branch, we have to delete its HEAD reference as shown here.

Deleting a branch
1

DELETE /repos/octocat/Hello-World/git/refs/heads/feature-a

More tips on Github API

1) When processing data from Github API, note that any commit has an author and a committer, as shown below.

Example commit data
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

    "commit": {
        "author": {
            "name": "Cuong Dong-Si",
            "email": "tdongsi@example.com",
            "date": "2017-08-17T05:33:46Z"
        },
        "committer": {
            "name": "Tue-Cuong Dong-Si",
            "email": "tdongsi@example.com",
            "date": "2017-08-17T05:33:46Z"
        },
        "message": "@JIRA-4214772@: Add function.",
        "tree": {
            "sha": "xxx",
            "url": "xxx"
        },
        "url": "xxx",
        "comment_count": 0
    },

While the two fields are usually the same in normal commits (with same associated email and timestamp), they have different meanings. In summary, the author is the one who created the content, and the committer is the one who committed it. The two fields can be different in some common Github workflows:

  • Commit a change from Github web interface: The author is the logged-in user (e.g., tdongsi) but the “committer” field usually has the Github default name and email, e.g., “Github Enterprise” and “no-reply@github.com”.
  • Make and/or merge a pull request from Github: For example, Alice submitted a pull request which was accepted and then merged by Betty (the repository owner). In that case, the author is Alice and the committer is Betty.

Due to that subtle difference in committer and author in different scenarios, one has to be careful when using data sent by Github API in a Jenkins pipeline. For example, you want to send email to the repository owner (committer) at the end of a Pull Request build, but what if someone adds a commit via Github web interface (commiter email would be “no-reply@github.com” which is not helpful).

2) There is an API rate limit for the free public Github API (note “X-RateLimit-Limit” and “X-RateLimit-Remaining” in output below).

Github API limit
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

tdongsi-mac:dev tdongsi$ curl -i https://api.github.com/users/tdongsi
HTTP/1.1 200 OK
Date: Fri, 09 Jun 2017 16:16:49 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1236
Server: GitHub.com
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 55
X-RateLimit-Reset: 1497025098
Cache-Control: public, max-age=60, s-maxage=60
Vary: Accept
ETag: "4d7770cf5c2478bf64d23bc908494172"
Last-Modified: Thu, 01 Jun 2017 01:09:00 GMT
X-GitHub-Media-Type: github.v3; format=json
Access-Control-Expose-Headers: ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval
Access-Control-Allow-Origin: *
Content-Security-Policy: default-src 'none'
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
X-Runtime-rack: 0.030687
Vary: Accept-Encoding
X-Served-By: 62cdcc2d03a2f173f1c58590d1a90077
Vary: Accept-Encoding
X-GitHub-Request-Id: FADF:2CB6E:44F743B:56EDC6F:593AC9F1

...

You are likely to hit this rate limit quickly if you are polling the repos for updates. Instead of polling from your CI (e.g., Jenkins) system, it is recommended to use Github webhooks.

Reference

]]> <![CDATA[Jsonnet Code Recipes]]> 2018-07-12T14:24:32-07:00 http://tdongsi.github.io/blog/2018/07/12/jsonnet-snippets If you are sending/working with lots of JSON data in files such as AWS CloudFormation templates, Jsonnet tool can help reducing the hassle of maintaining. Using Jsonnet templates, it is easier to organize data and reduce repeated code present in such JSON data. This post goes over a few common Jsonnet code recipes for generating JSON data.

Automated tests

At least, make sure your jsonnet template files can compile. The following example bash script will find all the manifest files and try to compile that:

1
2
3
4
5
6
7
8
9
10

# Provide a superset of required parameters
for i in `find Project1/manifests Project2/manifests   -name "*.jsonnet"`;
do
  jsonnet -V param1=1 -V param2=dummy -V param3=1 "${i}" >> /dev/null
done;
# This will lint-test the files, including libsonnet files.
for i in `find Project1 Project2 -name "*.*sonnet"`;
do
  jsonnet fmt -i -n 2 "${i}" --test
done;

Conditional add

Conditionally adding items to a list.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

  ports: [
           {
             port: $.httpPort,
             targetport: $.httpTargetPort,
           },
           {
             port: $.jnlpPort,
             targetport: 15372,
           },
         ] +
         (if $.sshEnabled then
            [{
              port: $.sshPort,
              targetport: 15373,
            }] else []),

Conditionally adding attributes to an object/map.

1
2
3
4
5
6
7

  defaultContainerEnv:: {
    LOGGING_STDERR_LEVEL: "ALL",
    JENKINS_USER: "jenkins",
  } + (if $.sshEnabled then {
          SSH_PORT: 7012,
        } else {}
      )
]]> <![CDATA[Jenkins Pipeline Unit Testing]]> 2018-06-07T22:33:46-07:00 http://tdongsi.github.io/blog/2018/06/07/jenkins-pipeline-unit-testing Jenkins shared library is a powerful way for sharing Groovy code between multiple Jenkins pipelines. However, when many Jenkins pipelines, including mission-critical deployment pipelines, depend on such shared libraries, automated testing becomes necessary to prevent regressions whenever new changes are introduced into shared librariers. Despite its drawbacks, the third-party Pipeline Unit Testing framework satisfies some of automated testing needs. It would allow you to do mock execution of pipeline steps and checking for expected behaviors before actually running in Jenkins. However, documentation for this third-party framework is severely lacking (mentioned briefly here) and it is one of many reasons that unit testing for Jenkins shared libraries is usually an after-thought, instead of being integrated early. In this blog post, we will see how to do unit testing for Jenkins shared library with the Pipeline Unit Testing framework.

Testing Jenkins shared library

Example Groovy file

For this tutorial, we look at the following Groovy build wrapper as the example under test:

buildWrapper.groovy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

def call(Closure body) {
  def config = [:]

  if (body != null) {
    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()
  }

  def settings = config.settings ?: "settings.xml"

  node('java-agent') {
    stage('Checkout') {
      checkout scm
    }

    stage('Main') {
      // Test Python setup
      sh(script: 'python -c "import requests"', returnStatus: true)
      // Test Docker setup
      sh 'docker version'
    }

    stage('Post') {
      // Print info of standard tools
      sh 'ls -al'
      sh 'java -version'
      sh "mvn -s $settings -version"
      sh 'python -V'
    }
  }
}

After the shared library is set up properly, you can call the above Groovy build wrapper in Jenkinsfile as follows to use default parameters:

Jenkinsfile for first use case 
1
2

buildWrapper {
}

or you can set the parameters in the wrapper’s body as follows:

Jenkinsfile for second use case
1
2
3

buildWrapper {
  settings = "dummy.xml"
}

In the next section, we will look into automated testing of both use cases using JenkinsPipelineUnit.

Using JenkinsPipelineUnit

To use JenkinsPipelineUnit, it is recommended to set up IntelliJ following this tutorial.

To test the above buildWrapper.groovy using the Jenkins Pipeline Unit, you can start with a unit test for the second use case as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

  /**
   * Represent the call:
   *     buildWrapper {
   *       settings = "dummy.xml"
   *     }
   *
   * @throws Exception
   */
  @Test
  public void configured() throws Exception {
    def script = loadScript('vars/buildWrapper.groovy')
    script.call({
      settings = "dummy.xml"
    })

    printCallStack()
  }

Unfortunately, when executing that unit test, it is very likely that you will get various errors that are not well-explained by JenkinsPipelineUnit documentation.

Stack trace
1
2
3
4
5
6

groovy.lang.MissingPropertyException: No such property: scm for class: demoWrapper

  at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:66)
  at org.codehaus.groovy.runtime.callsite.PogoGetPropertySite.getProperty(PogoGetPropertySite.java:51)
  at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSite.java:310)
  at demoWrapper$_call_closure1$_closure2.doCall(demoWrapper.groovy:19)

The short explanation is that the mock execution environment is not properly set up. First, we need to call setUp() from the base class BaseRegressionTest of JenkinsPipelineUnit to set up the mock execution environment. In addition, since most Groovy scripts will have this statement checkout scm, we need to mock the Jenkins global variable scm, which represents the SCM state (e.g., Git commit) associated with the current Jenkinsfile. The most simple way to mock it is to set it to empty state as follows:

Mocking Jenkins variable scm
1

binding.setVariable('scm', [:])

We can also set it to a more meaningful value such as a Git branch as follows:

Mocking Jenkins variable scm
1
2
3
4
5
6
7
8

binding.setVariable('scm', [
    $class: 'GitSCM',
    branches: [[name: 'master']],
    doGenerateSubmoduleConfigruations: false,
    extensions: [],
    submoduleCfg: [],
    userRemoteConfigs: [[url: "/var/git-repo"]]
])

However, an empty scm will usually suffice. Besides Jenkins variables, we can also register different Jenkins steps/commands as follows:

Mocking Jenkins step library
1

helper.registerAllowedMethod('library', [String.class], null)

After going through the setup steps above, you should have the following setup method like this:

Minimum setup method
1
2
3
4
5
6
7
8
9
10
11
12
13

import com.lesfurets.jenkins.unit.BaseRegressionTest

class DemoTest extends BaseRegressionTest {

  @Override
  @Before
  public void setUp() throws Exception {
    super.setUp();
    binding.setVariable('scm', [:])
    helper.registerAllowedMethod('library', [String.class], null)
  }
...
}

Rerunning the above unit test will show the full stack of execution:

1
2
3
4
5
6
7
8
9
10
11
12

   buildWrapper.call(groovy.lang.Closure)
      buildWrapper.node(java-agent, groovy.lang.Closure)
         buildWrapper.stage(Checkout, groovy.lang.Closure)
            buildWrapper.checkout({})
         buildWrapper.stage(Main, groovy.lang.Closure)
            buildWrapper.sh({script=python -c "import requests", returnStatus=true})
            buildWrapper.sh(docker version)
         buildWrapper.stage(Post, groovy.lang.Closure)
            buildWrapper.sh(ls -al)
            buildWrapper.sh(java -version)
            buildWrapper.sh(mvn -s dummy.xml -version)
            buildWrapper.sh(python -V)

For automated detection of regression, we need to save the expected call stack above into a file into a location known to JenkinsPipelineUnit. You can specify the location of such call stacks by overriding the field callStackPath of BaseRegressionTest in setUp method. The file name should follow the convention ${ClassName}_${subname}.txt where subname is specified by testNonRegression method in each test case. Then, you can update the above test case to perform regression check as follows:

1
2
3
4
5
6
7
8
9
10

  @Test
  public void configured() throws Exception {
    def script = loadScript('vars/demoWrapper.groovy')
    script.call({
      settings = "dummy.xml"
    })

    // printCallStack()
    testNonRegression("configured")
  }

In this example, the above call stack should be saved into DemoTest_configured.txt file at the location specified by callStackPath. Similarly, you can also have another unit test for the other use case of buildWrapper.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

  /**
   * Represent the call:
   * buildWrapper {
   * }
   *
   * @throws Exception
   */
  @Test
  public void default_value() throws Exception {
    def script = loadScript('vars/buildWrapper.groovy')
    script.call({})

    // printCallStack()
    testNonRegression("default")
  }

Any change in buildWrapper.groovy will be detected as test failures, as shown in the screen shot below.

Screeshot

In IntelliJ, we can click on Click to see difference link to compare the actual call stack versus the expected one that was saved in the text file.

Screeshot

This test class shows a complete example, together with files of expected call stacks.

Other usage

You can also use PipelineUnitTests to test Jenkinsfile. In most cases, testing Jenkinsfile will be similar to testing Groovy files in vars folder, as explained above, since they are quite similar.

Example Jenkinsfile
1
2
3
4
5
6
7
8
9
10
11

node() {
  stage('Checkout') {
    checkout scm
    sh 'git clean -xdf'
  }

  stage('Build and test') {
    sh './gradlew build'
    junit 'build/test-results/test/*.xml'
  }
}

The process is very similar: you need to mock out some global variables and functions corresponding to Jenkins pipeline steps. You will need to printCallStack to obtain the expected output and save it into some text file. Then, you can use testNonRegression for automated verification of no-regression in Jenkinsfile. This test class shows an example of testing Jenkinsfile using PipelineUnitTests.

Note that, unlike Groovy files in vars folder, Jenkinsfiles are regularly updated and usually NOT depended/used by any other codes. Therefore, automated tests for Jenkinsfile are not very common because of the cost/effort required.

References

]]> <![CDATA[IntelliJ Setup for Jenkins Development]]> 2018-02-09T21:32:59-08:00 http://tdongsi.github.io/blog/2018/02/09/intellij-setup-for-jenkins-shared-library-development This posts will show how to setup IntelliJ for development of Jenkins Groovy Init Scripts and shared libraries, including auto-complete for Jenkins pipeline steps. This is based on my original write-up in this project.

NOTE: this setup is NOT intended for Jenkins plugin or core development.

Start a new Gradle project

It is best to start a new project:

  1. Select File | New Project
  2. Select Gradle
  3. Select Java AND Groovy Screeshot
  4. Choose GroupId and ArtifactId Screeshot
  5. Enter path to Gradle. For Gradle on Mac installed via Homebrew, the Gradle home is like this: Screeshot NOTE: For Gradle installed on a Mac via Homebrew, the path “/usr/local/opt/gradle/libexec” may be preferrable to “/usr/local/Cellar/gradle/X.X/libexec” since the former will stay the same after Gradle version upgrades. In addition, if you work extensively with Grails/Gradle/Groovy, you may prefer installing via sdk tool instead of Homebrew.
  6. Choose Project name and Project Location Screeshot
  7. Finish Screeshot

Configure IDEA

Set up for Jenkins Plugins files which are of types .hpi or .jpi.

  1. Select IntelliJ IDEA | Preferences | Editor | File Types
  2. Select Archive
  3. Select + at the bottom left corner
  4. Add both .hpi and .jpi
  5. Select OK

Screeshot

Modify build.gradle to add the following lines.

1
2
3
4
5
6
7
8
9
10

compile 'org.jenkins-ci.main:jenkins-core:2.23'

// Jenkins plugins
compile group: 'org.jenkins-ci.plugins', name: 'credentials', version: '2.1.13', ext: 'jar'
compile group: 'org.jenkins-ci.plugins', name: 'matrix-auth', version: '1.6', ext: 'jar'
compile group: 'org.jenkins-ci.plugins.workflow', name: 'workflow-cps', version: '2.39', ext: 'jar'

// TRICKY: The lib folder contains all other plugins *JAR* files
// if not found in Maven
compile fileTree(dir: 'lib', include: ['*.jar'])

The above example will grab Jenkins core libraries, Matrix Authorization Plugin hpi, other plugin dependencies and javadocs for all imported libraries. Having these libraries imported will enable code auto-completion, syntax checks, easy refactoring when working with Groovy scripts for Jenkins. It will be a great productivity boost.

NOTE 1: The last line compile fileTree is the last resort for any Jenkins plugins that you cannot find the right group ID and artifact ID. It is rare these days but such cases cannot be completely ruled out.

NOTE 2: The ext: 'jar' is VERY important to ensure that jar files, instead of hpi/jpi files, are being downloaded and understood by IntellJ. Without that ext option specified, IntellJ won’t find JAR files nested in hpi/jpi files which is the default binaries for Jenkins plugins.

The final build.gradle will look like this. All of the above setup should suffice for working with Groovy Init Scripts. For working with Jenkins Shared Pipeline Libraries, we should take one extra step shown as follows.

Setup for Jenkins pipeline library

All Groovy files in Jenkins shared library for Pipelines have to follow this directory structure:

Directory structure of a Shared Library repository
1
2
3
4
5
6
7
8
9
10
11
12

(root)
+- src                     # Groovy source files
|   +- org
|       +- foo
|           +- Bar.groovy  # for org.foo.Bar class
+- vars
|   +- foo.groovy          # for global 'foo' variable
|   +- foo.txt             # help for 'foo' variable
+- resources               # resource files (external libraries only)
|   +- org
|       +- foo
|           +- bar.json    # static helper data for org.foo.Bar

Note that the Groovy code can be in both src and vars folders. Therefore, you need to add the following lines in build.gradle to inform Gradle locations of Groovy source codes:

1
2
3
4
5
6
7
8
9
10
11
12
13

sourceSets {
    main {
        groovy {
            srcDirs = ['vars', 'src']
        }
    }

    test {
        groovy {
            srcDirs = ['test']
        }
    }
}

Optionally, for unit testing Jenkins shared library, we have to add the following dependencies into our build.gradle file.

1
2

testCompile group: 'com.lesfurets', name: 'jenkins-pipeline-unit', version: '1.1'
testCompile group: 'org.spockframework', name: 'spock-core', version: '1.1-groovy-2.4'

Please see this blog post for more details on unit testing. The final build.gradle will look like this.

Auto-completion for Jenkins Pipeline

IntelliJ can’t auto-complete Jenkins pipeline steps such as echo or sh out of the box. We have to make it aware of those Jenkins pipeline DSLs, via a generic process explained here. Fortunately, it is much easier than it looks and you don’t have to actually write GroovyDSL script for tens of Jenkins pipeline steps. Jenkins make it easy by auto-generating the GroovyDSL script and it is accessible via “IntelliJ IDEA GDSL” link, as shown in screenshot below.

Screeshot

The “IntelliJ IDEA GDSL” link can be found by accessing “Pipeline Syntax” section, which is visible in the left navigation menu of any Pipeline-based job (e.g., “Admin” job in the example above). After clicking on the “IntelliJ IDEA GDSL” link, you will be able to download a plain text file with content starting like this:

IntelliJ IDEA GDSL
1
2
3
4
5
6
7
8

//The global script scope
def ctx = context(scope: scriptScope())
contributor(ctx) {
method(name: 'build', type: 'Object', params: [job:'java.lang.String'], doc: 'Build a job')
method(name: 'build', type: 'Object', namedParams: [parameter(name: 'job', type: 'java.lang.String'), parameter(name: 'parameters', type: 'Map'), parameter(name: 'propagate', type: 'boolean'), parameter(name: 'quietPeriod', type: 'java.lang.Integer'), parameter(name: 'wait', type: 'boolean'), ], doc: 'Build a job')
method(name: 'echo', type: 'Object', params: [message:'java.lang.String'], doc: 'Print Message')
method(name: 'error', type: 'Object', params: [message:'java.lang.String'], doc: 'Error signal')
...

As you can see, it is a GroovyDSL file that describes the known pipeline steps such as echo and error. Note that GDSL files can be different for different Jenkins instances, depending on Pipeline-supported plugins currently installed on individual Jenkins instance. To make IntelliJ aware of the current Jenkins pipeline steps available on our Jenkins, we need to place that GDSL file into a location known to source folders. As shown in the last section, anywhere in both vars and src folders are eligible as such although I personally prefer to put the GDSL file into vars folder (for example).

After installing the GDSL file into a proper location, IntelliJ may complain with the following message DSL descriptor file has been change and isn’t currently executed and you have to click Activate back to get the IntelliJ aware of the current DSLs. After that, you can enjoy auto-completion as well as documentation of the Jenkine Pipeline DSLs.

More information

]]> <![CDATA[Design Patterns for Container-based Distributed Systems]]> 2018-01-14T14:56:56-08:00 http://tdongsi.github.io/blog/2018/01/14/design-patterns-for-container-based-distributed-systems In early and mid-1990s, object-oriented programming (OOP) revolutionized software development by dividing applications into modular components as objects and classes. Today, the rise of microservice architectures and containerization technologies enable a similar revolution in developing distributed systems and SaaS/PaaS products. “Containers” and “container images” are analogus to “objects” and “classes” in OOP, respectively, as a unit of development and deployment in distributed systems. In this paper, the authors argued that just like Design Patterns for Object-Oriented Programming, codified in the famous “Gang of Four” book, some Design Patterns have emerged for Container-based Distributed Systems. In the same paper, the authors attempted to do something similar, documenting the most common patterns and their usage.

General ideas

The container and the container image should be the abstractions for the development of distributed systems. Similar to what objects and classes did for OOP, thinking in term of containers abstracts away the low-level details of code and allows us to think in higher-level design patterns. Based on how containers interact with other containers and get deployed into actual underlying machines, the authors divide the patterns in to three main groups:

  • Single-container patterns: How to expose interface of application in container (similar to effective design of object interface).
    • Upward direction: expose application info/metrics such as /health endpoint.
    • Downward direction: Formal life cycle agreed between application and management system (similar to Android Activity model).
  • Single-node multi-container patterns: Basically, how to design a pod in Kubernetes (pod = group of symbiotic containers)
    • Sidecar pattern:
      • Sidecar containers will extend and enhance the main container.
      • Example: Log forwarding sidecar that collects logs from main container from local disk and stream to a cluster storage system.
    • Ambassador pattern:
      • Ambassador containers will proxy communication to and from the main container.
      • Ambassador simplifies and standardizes the outside world to the main container.
      • Example: Redis proxy ambassador that will discover dependent services for the main container.
    • Adapter pattern:
      • Adapter containers will standardize and normalize the output of the main container.
      • In contrast to ambassador, adapter simplifies and normalizes the main app to outside world.
      • Example: Adapters to ensure all containers have the same monitoring interface to hook to central monitoring system.
  • Multi-node patterns
    • Leader election pattern
      • When we have many replicas, but only one of them is active at a time.
    • Work queue pattern
      • One coordinator and many workers distributed to as many nodes for processing.
    • Scatter/Gather pattern
      • Similar to “Work queue” pattern, except one coordinator scatter partial works to many slaves, then gather/merge partial outcomes from those slaves.
      • Example: A search engine service (e.g., Google) will multicast a request (search query) to all workers. Each worker will compute a local result and send it back to a gatherer who will consolidate into a single presentable response.

Why multiple containers on single node

The more contentious patterns are probably single-node multi-container patterns, especially the sidecar pattern. The most common anti-pattern is that we try to merge the functionality of the sidecar container into the main container. Analogously, we also have seen a similar anti-pattern in OOP that ends up with a large class that tries to do many things at once. There are several benefits to use separate containers:

  • Container is the unit of resource allocation and accounting.
    • In the sidecar example above, the log forwarding container is configured to scavenge spare CPU cycles when the web server is not busy.
  • Container is the unit of packaging.
    • Two separate teams can work independently on log forwarding and web application.
  • Container is the unit of reuse.
    • The log forwarding sidecar image can be paired with other “main” containers.
  • Container provides failure containment boundary.
    • If the log forwarding container fails, the application container continues serving.
  • Container is the unit of deployment.
    • Each component can be functionally upgraded or rolled back independently.

References

]]> <![CDATA[Groovy Hook Script and Jenkins Configuration as Code]]> 2017-12-30T21:02:48-08:00 http://tdongsi.github.io/blog/2017/12/30/groovy-hook-script-and-jenkins-configuration-as-code This post discusses Groovy Hook Scripts and how to use them for full configuration-as-code in Jenkins with Docker, Pipeline. This can help us to set up local environment for developing Jenkins Pipeline libraries and to evaluate various Jenkins features.

Groovy Hook Scripts

These scripts are written in Groovy, and get executed inside the same JVM as Jenkins, allowing full access to the domain model of Jenkins. For a given hook HOOK, the following locations are searched:

1
2
3
4

WEB-INF/HOOK.groovy in jenkins.war
WEB-INF/HOOK.groovy.d/*.groovy in the lexical order in jenkins.war
$JENKINS_HOME/HOOK.groovy
$JENKINS_HOME/HOOK.groovy.d/*.groovy in the lexical order

The init is the most commonly used hook (i.e., HOOK=init). The following sections show how some of the most common tasks and configurations in Jenkins can be achieved by using such Groovy scripts. For example, in this project, many of such scripts are added into a Dockerized Jenkins master and executed when starting a container to replicate configurations of the Jenkins instance in production. It will give us ability to quickly spin up local Jenkins instances for development or troubleshooting issues in production Jenkins.

On a side note, IntelliJ IDEA is probably the best development tool for working with these Groovy Scripts. Check out these instructions on how to set it up in IntelliJ. UPDATED ON 2018/09/29: More on IntelliJ setup is discussed in this blog post.

Authorization

This section shows how to enable different authorization strategies in Groovy code.

"Logged-in users can do anything"
1
2
3
4
5
6
7
8
9
10
11
12

import jenkins.model.*
def instance = Jenkins.getInstance()

import hudson.security.*
def realm = new HudsonPrivateSecurityRealm(false)
instance.setSecurityRealm(realm)

def strategy = new hudson.security.FullControlOnceLoggedInAuthorizationStrategy()
strategy.setAllowAnonymousRead(false)
instance.setAuthorizationStrategy(strategy)

instance.save()

Matrix-based authorization: Gives all authenticated users admin access:

Matrix-based authorization
1
2
3
4
5
6
7
8
9
10
11
12

import jenkins.model.*
def instance = Jenkins.getInstance()

import hudson.security.*
def realm = new HudsonPrivateSecurityRealm(false)
instance.setSecurityRealm(realm)

def strategy = new hudson.security.GlobalMatrixAuthorizationStrategy()
strategy.add(Jenkins.ADMINISTER, 'authenticated')
instance.setAuthorizationStrategy(strategy)

instance.save()

For importing GlobalMatrixAuthorizationStrategy class, make sure that matrix-auth plugin is installed. For full list of standard permissions in the matrix, see this code snippet. Note that the matrix can be different if different plugins are installed. For example, the “Replay” permission for Runs is not simply hudson.model.Run.REPLAY since there is no such static constant. Such permission is only available after Workflow CPS plugin is installed. Therefore, we can only set “Replay” permission for Runs with the following:

1

strategy.add(org.jenkinsci.plugins.workflow.cps.replay.ReplayAction.REPLAY,USER)

References

Basic Jenkins security

In addition to enable authorization strategy, we should also set some basic configurations for hardening Jenkins. Those includes various options that you see in Jenkins UI when going to Manage Jenkins > Configure Global Security.

  • Disable Jenkins CLI
  • Limit Jenkins agent protocols.
  • “Enable Slave -> Master Access Control”
  • “Prevent Cross Site Request Forgery exploits”
Basic Jenkins security
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

import hudson.security.csrf.DefaultCrumbIssuer
import jenkins.model.Jenkins
import jenkins.model.JenkinsLocationConfiguration
import jenkins.security.s2m.AdminWhitelistRule
import org.kohsuke.stapler.StaplerProxy
import hudson.tasks.Mailer

println("--- Configuring Remoting (JNLP4 only, no Remoting CLI)")
Jenkins.instance.getDescriptor("jenkins.CLI").get().setEnabled(false)
Jenkins.instance.agentProtocols = new HashSet<String>(["JNLP4-connect"])

println("--- Enable Slave -> Master Access Control")
Jenkins.instance.getExtensionList(StaplerProxy.class)
    .get(AdminWhitelistRule.class)
    .masterKillSwitch = false

println("--- Checking the CSRF protection")
if (Jenkins.instance.crumbIssuer == null) {
    println "CSRF protection is disabled, Enabling the default Crumb Issuer"
    Jenkins.instance.crumbIssuer = new DefaultCrumbIssuer(true)
}

println("--- Configuring Quiet Period")
// We do not wait for anything
Jenkins.instance.quietPeriod = 0
Jenkins.instance.save()

println("--- Configuring Email global settings")
JenkinsLocationConfiguration.get().adminAddress = "admin@non.existent.email"
Mailer.descriptor().defaultSuffix = "@non.existent.email"

Some are not working for versions before 2.46, according to this. For disabling Jenkins CLI, you can simply add the java argument -Djenkins.CLI.disabled=true on Jenkins startup.

References

Create Jobs and Items

Create "Pipeline script from SCM" job
1
2
3
4
5
6
7
8
9
10

import hudson.plugins.git.*;

def scm = new GitSCM("git@github.com:dermeister0/Tests.git")
scm.branches = [new BranchSpec("*/develop")];

def flowDefinition = new org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition(scm, "Jenkinsfile")

def parent = Jenkins.instance
def job = new org.jenkinsci.plugins.workflow.job.WorkflowJob(parent, "New Job")
job.definition = flowDefinition

Create different kinds of Credentials

Adding Credentials to a new, local Jenkins for development or troubleshooting can be a daunting task. However, with the following scripts and the right setup (NEVER commit your secrets into VCS), developers can automate adding the required Credentials into the new Jenkins.

Preamble
1
2
3
4
5
6
7
8
9
10
11
12
13

import com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl
import org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl
import org.jenkinsci.plugins.plaincredentials.impl.FileCredentialsImpl
import com.cloudbees.plugins.credentials.domains.Domain
import com.cloudbees.plugins.credentials.CredentialsScope
import jenkins.model.Jenkins
import hudson.util.Secret
import com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey
import com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl
import com.cloudbees.plugins.credentials.SecretBytes

def domain = Domain.global()
def store = Jenkins.instance.getExtensionList('com.cloudbees.plugins.credentials.SystemCredentialsProvider')[0].getStore()
"Username with Password" type
1
2
3
4
5
6

def githubAccount = new UsernamePasswordCredentialsImpl(
        CredentialsScope.GLOBAL, "test-github", "Test Github Account",
        "testuser",
        "testpassword"
)
store.addCredentials(domain, githubAccount)
"Secret text" type
1
2
3
4
5

def secretString = new StringCredentialsImpl(
        CredentialsScope.GLOBAL, "test-secret-string", "Test Secret String",
        Secret.fromString("testpassword")
)
store.addCredentials(domain, secretString)
"Secret file" type
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

// Text file
def secret = '''Hi,
This is the content of the file.
'''

def secretBytes = SecretBytes.fromBytes(secret.getBytes())
def secretFile = new FileCredentialsImpl(
  CredentialsScope.GLOBAL,
  'text-secret-file',
  'description',
  'file.txt',
  secretBytes)
store.addCredentials(domain, secretFile)

// Binary file
Path fileLocation = Paths.get("/path/to/some/file.tar");
def secretBytes = SecretBytes.fromBytes(Files.readAllBytes(fileLocation))
def secretFile = new FileCredentialsImpl(
  CredentialsScope.GLOBAL,
  'binary-secret-file',
  'description',
  'file.tar',
  secretBytes)
store.addCredentials(domain, secretFile)
"SSH Username with private key" type
1
2
3
4
5
6
7
8
9
10

String keyfile = "/var/jenkins_home/.ssh/id_rsa"
def privateKey = new BasicSSHUserPrivateKey(
        CredentialsScope.GLOBAL,
        "jenkins_ssh_key",
        "git",
        new BasicSSHUserPrivateKey.FileOnMasterPrivateKeySource(keyfile),
        "",
        ""
)
store.addCredentials(domain, privateKey)
"Certificate" type
1
2
3
4
5
6
7
8

String minikubeKeyfile = "/var/jenkins_home/secret_data/minikube.pfx"
def minikubeCreds = new CertificateCredentialsImpl(
        CredentialsScope.GLOBAL,
        "minikube",
        "Minikube client certificate",
        "secret",
        new CertificateCredentialsImpl.FileOnMasterKeyStoreSource(minikubeKeyfile))
store.addCredentials(domain, minikubeCreds)

Notifications

Configure Slack
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import jenkins.model.*
def instance = Jenkins.getInstance()

// configure slack
def slack = Jenkins.instance.getExtensionList(
  jenkins.plugins.slack.SlackNotifier.DescriptorImpl.class
)[0]
def params = [
  slackTeamDomain: "domain",
  slackToken: "token",
  slackRoom: "",
  slackBuildServerUrl: "$JENKINS_URL",
  slackSendAs: ""
]
def req = [
  getParameter: { name -> params[name] }
] as org.kohsuke.stapler.StaplerRequest
slack.configure(req, null)
slack.save()
Global email settings
1
2
3
4
5
6

import jenkins.model.*
def instance = Jenkins.getInstance()

// set email
def location_config = JenkinsLocationConfiguration.get()
location_config.setAdminAddress("jenkins@skynet.net")
]]> <![CDATA[`src` Folder in Jenkins Shared Library]]> 2017-12-26T11:18:09-08:00 http://tdongsi.github.io/blog/2017/12/26/class-in-jenkins-shared-library This post reviews best practices when we implement Groovy classes and/or static Groovy methods, in src folder as opposed to vars folder, for Jenkins Shared Library.

Examples of shared libraries in src folder

All Groovy files in Jenkins shared library for pipelines have to follow this directory structure:

Directory structure of a Shared Library repository
1
2
3
4
5
6
7
8
9
10
11
12

(root)
+- src                     # Groovy source files
|   +- org
|       +- foo
|           +- Bar.groovy  # for org.foo.Bar class
+- vars
|   +- foo.groovy          # for global 'foo' variable
|   +- foo.txt             # help for 'foo' variable
+- resources               # resource files (external libraries only)
|   +- org
|       +- foo
|           +- bar.json    # static helper data for org.foo.Bar

src folder is intended to set up with groovy files in the standard directory structure, such as “src/org/foo/bar.groovy”. It will be added to the class path when the Jenkins pipelines are executed.

Any custom function in a Jenkins shared library has to eventually use basic Pipeline steps such as sh or git, made available through various Jenkins plugins. However, Groovy classes in shared Jenkins library cannot simply call those basic steps directly. There are a few approaches on how to access those Pipeline steps indirectly.

Method 1: Methods in implicit class

Groovy scripts in src folder can implement methods that invoke Pipeline steps, like this:

Example 1
1
2
3
4
5
6

// src/org/demo/buildUtils.groovy
package org.demo

def checkOutFrom(repo) {
  git url: "git@github.com:jenkinsci/${repo}"
}

The method is stored implicitly in library and can then be invoked from a Scripted Pipeline like this:

Example 1 (continued)
1
2

def myUtils = new org.demo.buildUtils()
myUtils.checkOutFrom(repo)

However, the requirement of this approach is that those methods defined in buildUtils.groovy cannot be enclosed in any class. The “implicit class” mentioned in this approach refers to the fact that any Groovy script, such as buildUtils.groovy, has an implicit class (e.g., org.demo.buildUtils) that contains all the defined functions in it. This approach has limitations; for example, it prevents the declaration of a superclass.

Method 2: Explicit objects

In the following example, we create an enclosing class that would facilitate things like defining a superclass. In that case, to access standard DSL steps such as sh or git, we can explicitly pass special global variables env and steps into a constructor or a method of the class. Global object env contains all current environment variables while steps contains all standard pipeline steps. Note that the class must also implement Serializable interface to support saving the state if the pipeline is stopped or resumed.

Example 2
1
2
3
4
5
6
7
8
9
10
11
12
13

package org.demo
class Utilities implements Serializable {
  def env
  def steps
  Utilities(env, steps) {
    this.env = env
    this.steps = steps
  }

  def mvn(args) {
    steps.sh "${steps.tool 'Maven'}/bin/mvn -o ${args}"
  }
}
Example 2 (continued)
1
2
3
4
5
6
7

@Library('utils')
import org.foo.Utilities

def utils = new Utilities(env, steps)
node {
  utils.mvn 'clean package'
}

Method 3: Static methods in explicit class

In the final example, we can also use static method and pass in the script object, which already has access to everything, including environment variables script.env and Pipeline steps such as script.sh.

Example 3
1
2
3
4
5
6

package org.demo
class Utilities {
  static def mvn(script, args) {
    script.sh "${script.tool 'Maven'}/bin/mvn -s ${script.env.HOME}/jenkins.xml -o ${args}"
  }
}

The above example shows the script being passed in to one static method, invoked from a Scripted Pipeline as follows (note import static):

Example 3 (continued)
1
2
3
4
5

@Library('utils')
import static org.demo.Utilities.*
node {
  mvn this, 'clean package'
}

Recommended practices

All three approaches shown in three examples above are valid in Scripted Jenkinsfile. However, per recommended by CloudBees Inc., src folder is best for utility classes that contains a bunch of static Groovy methods. It is easier to use global variables in the vars directory instead of classes in the src directory, especially when you need to support declarative pipelines in your team. The reason is that in declarative pipelines, the custom functions in Jenkins shared libraries must be callable in declarative syntax, e.g., “myCustomFunction var1, var2” format. As you can see in the examples above, only in Method 3 (Static methods in explicit class), where custom functions are defined as static methods, the invocation of custom function is compatible with declarative pipeline syntax.

When using src area’s Groovy codes with library step, you should use a temporary variable to reduce its verbosity, as follows:

Reduce verbosity
1
2
3
4

def mvn = library('utils').org.demo.Utilities.mvn
mvn this, 'clean package'
// or 
// mvn self, 'clean package'

Reference

]]> <![CDATA[TCP Ping]]> 2017-10-09T14:36:56-07:00 http://tdongsi.github.io/blog/2017/10/09/tcp-ping This post explains a work-around for pinging hosts in network where ICMP requests are blocked.

There are times that we can ssh to our servers but simply can’t ping those servers.

ssh works but ping fails
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

tdongsi$ ping 10.252.158.71
PING 10.252.158.71 (10.252.158.71): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
^C
--- 10.252.158.71 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

tdongsi$ ssh centos@10.252.158.71
Last login: Mon Feb  6 23:35:41 2017 from 10.3.55.249
[centos@kube-1 ~]$ exit
logout
Connection to 10.252.158.71 closed.

One possible explanation of seemingly perplexing situation like above is that ICMP requests (i.e., ping) are blocked. It is not unheard of that an ISP or a network administrator blocks ICMP requests. To work around that limitation, you can use a “TCP ping” on a port, using a tool like nmap. The following examples check if a host can be reached via port 80:

"TCP ping" with nmap: success and failure.
1
2
3
4
5
6
7
8
9
10
11
12

tdongsi$ nmap -sn -PS80 10.252.158.71

Starting Nmap 7.40 ( https://nmap.org ) at 2017-03-03 17:17 PST
Nmap scan report for master-1 (10.252.158.71)
Host is up (0.051s latency).
Nmap done: 1 IP address (1 host up) scanned in 0.06 seconds

tdongsi$ nmap -sn -PS80 master-2

Starting Nmap 7.40 ( https://nmap.org ) at 2017-10-09 14:35 PDT
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 2.02 seconds
]]> <![CDATA[Jacoco in Maven Projects]]> 2017-09-23T21:39:13-07:00 http://tdongsi.github.io/blog/2017/09/23/jacoco-in-maven-project This blog post goes over some recipes for adding code coverage report to Maven-based projects with Jacoco.

Standard usage

Based on offical instruction and this, you need to add the following code snippet in to your Maven pom.xml.

Jacoco usage (typical Maven project)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

<project>
...

    <dependencies>
        ...
    </dependencies>

    <build>
        <plugins>
            ...
            <!-- Code Coverage report generation -->
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.7.9</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>generate-code-coverage-report</id>
                        <phase>test</phase>
                        <goals>
                            <goal>report</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

At least, you need “prepare-agent” before test phase for Jacoco instrumentation and “report” after test phase for generating the report. You could subject the project to code coverage and generate the same report without making any changes to the pom file. To do this, run the following command:

Jacoco from Maven command-line
1

mvn jacoco:prepare-agent test jacoco:report

You may get the following error:

1

[ERROR] No plugin found for prefix 'jacoco' in the current project ...

There are two options to fix that error. The easiest way is to specify the groupId and artifactId parameters of the plugin explicitly. You can also add version to ensure the stability of your build pipeline.

1

mvn clean org.jacoco:jacoco-maven-plugin:0.7.9:prepare-agent install org.jacoco:jacoco-maven-plugin:0.7.9:report

The more long-term solution is to add the following in to your Maven “settings.xml”.

Maven settings
1
2
3

<pluginGroups>
    <pluginGroup>org.jacoco</pluginGroup>
</pluginGroups>

Tests with Mock

If mocking is involved in unit tests, you need to use “instrument” and “restore-instrumented” steps.

Reference:

Multi-module Maven projects

Officially, multi-module Maven projects are supported differently by Jacoco as documented here. Instrumentation will be similar but the challenge of multi-module Maven projects lies in how to collect and report code coverage of all modules correctly. Jacoco Maven standard goals, as shown in sections above, work on single modules only: Tests are executed within the module and contributed coverage only to code within the same module. Coverage reports were created for each module separately.

In the past, there are some ad-hoc solutions such as this (for Jacoco 0.5.x) to work around that limit. However, those patterns are also error-prone and hard to customize, especially when Jacoco is used with Surefire plugin. Fortunately, Jacoco recently introduced a new Maven goal “report-aggregate” in its release 0.7.7 which will aggregate code coverage data across Maven modules. Its usage is also present in the same link (quoted below) but it is too succint and not very helpful for new users.

Create a dedicated module in your project for generation of the report. This module should depend on all or some other modules in the project.

Let' say you have a multi-module Maven project with this structure:

Multi-module Maven project
1
2
3
4

root pom
  |- module a
  |- module b
  |- module c

To use Jacoco “report-aggregate” goal for these modules, you first need to add a dedicated “coverage” module. This “coverage” module should be added into the root POM. The multi-module Maven project should now look like this:

Multi-module Maven project with aggregate coverage module
1
2
3
4
5

root pom
  |- module a
  |- module b
  |- module c
  |- module "coverage"

The POMs for each module does not need to change at all. The POM for the “coverage” module will look like this:

Maven pom.xml for coverage module
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>com.company</groupId>
        <artifactId>company-pom</artifactId>
        <version>3.0</version>
    </parent>

    <artifactId>report</artifactId>
    <name>Jacoco Report</name>

    <dependencies>
        <dependency>
            <groupId>my.example</groupId>
            <artifactId>module-a</artifactId>
            <version>210.0.00-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>my.example</groupId>
            <artifactId>module-b</artifactId>
            <version>210.0.00-SNAPSHOT</version>
        </dependency>
        <dependency>
            <groupId>my.example</groupId>
            <artifactId>module-c</artifactId>
            <version>210.0.00-SNAPSHOT</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.7.9</version>
                <configuration>
                    <excludes>
                        <!-- Example of excluding classes 
                        <exclude>**/com/company/config/AutoConfiguration.class</exclude>
                        -->
                    </excludes>
                </configuration>
                <executions>
                    <execution>
                        <id>report-aggregate</id>
                        <phase>verify</phase>
                        <goals>
                            <goal>report-aggregate</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <!-- This coverage module should never be deployed -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-deploy-plugin</artifactId>
                <configuration>
                    <skip>true</skip>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Note that we still require “prepare-agent” step to run before the first test suite. Depending on what plugins are being used and how the modules are organized within the project, we might have different setup for that particular step. One option is to run from the command-line:

Maven pom.xml for coverage module
1
2
3
4
5

tdongsi$ ls
Dockerfile              README.md               module1       pom.xml
Jenkinsfile             coverage                module2       scripts

tdongsi$ mvn -B clean org.jacoco:jacoco-maven-plugin:0.7.9:prepare-agent install

Links:

Customizations

In theory, a global threshold can be defined in coverage/pom.xml to enforce code coverage standard across teams. However, in practice, different teams are at different stages of module/service maturity and blindly having a global threshold will hamper teams working on newer services/modules. In addition, it does not make sense to enforce code coverage on some Maven modules such as those generated in GRPC.

In Jacoco, you can set different coverage limits for individual modules instead of a global threshold for all modules. In the following example, you can specify a coverage threshold for module A by modifying module A’s pom.xml file:

Module A's pom.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

...

</plugins>
...
    <plugin>
        <groupId>com.atlassian.maven.plugins</groupId>
        <artifactId>maven-clover2-plugin</artifactId>
    </plugin>

    <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>0.7.9</version>

        <executions>
            <execution>
                <id>check</id>
                <goals>
                <goal>check</goal>
                </goals>
                <configuration>
                    <!-- NOTE: Set haltOnFailureto true when code coverage is enforced -->
                    <haltOnFailure>false</haltOnFailure>

                    <rules>
                        <rule >
                            <element>CLASS</element>
                            <limits>
                                <limit >
                                    <counter>LINE</counter>
                                    <value>COVEREDRATIO</value>
                                    <minimum>0.80</minimum>
                                </limit>
                                <limit >
                                    <counter>BRANCH</counter>
                                    <value>COVEREDRATIO</value>
                                    <minimum>0.80</minimum>
                                </limit>
                            </limits>
                            <excludes>
                                <!-- 
                                <exclude>com.test.ExampleExclusion</exclude>
                                -->
                            </excludes>
                        </rule>
                    </rules>
                </configuration>
            </execution>
        </executions>
    </plugin>
</plugins>

As you can see, you can also specify files being excluded from coverage calculation.

References

]]> <![CDATA[Keep Gruntfile Clean With Load-grunt-config]]> 2017-09-15T23:29:17-07:00 http://tdongsi.github.io/blog/2017/09/15/keep-gruntfile-clean-with-load-grunt-config In this post, we look into how to keep our Gruntfile clean and tidy. By keeping our Gruntfile clean and tidy, it is easier for us to refine and improve the Grunt build process with its numerous plugins.

Starting point

Let’s say you start a new Node project.

Starting a Node project
1
2
3
4
5
6
7
8
9
10
11
12
13

# Init by creating package.json file
npm init
# Answer questions to create package.json file

# Adding grunt
npm install grunt --save-dev
npm install grunt-jslint --save-dev
npm install load-grunt-tasks --save-dev

# Initalizing Gruntfile
npm install grunt-init -g
grunt-init gruntfile
# grunt-init node

At the end of these steps, you have a basic package.json and Gruntfile. The basic Gruntfile would appear like this:

Basic Gruntfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

/*global module:false*/
module.exports = function(grunt) {

  // Project configuration.
  grunt.initConfig({
    // Metadata.
    pkg: grunt.file.readJSON('package.json'),
    banner: '/*! <%= pkg.title || pkg.name %> - v<%= pkg.version %> - ' +
      '<%= grunt.template.today("yyyy-mm-dd") %>\n' +
      '<%= pkg.homepage ? "* " + pkg.homepage + "\\n" : "" %>' +
      '* Copyright (c) <%= grunt.template.today("yyyy") %> <%= pkg.author.name %>;' +
      ' Licensed <%= _.pluck(pkg.licenses, "type").join(", ") %> */\n',
    // Task configuration.
    concat: {
      options: {
        banner: '<%= banner %>',
        stripBanners: true
      },
      dist: {
        src: ['lib/<%= pkg.name %>.js'],
        dest: 'dist/<%= pkg.name %>.js'
      }
    },
    uglify: {
      options: {
        banner: '<%= banner %>'
      },
      dist: {
        src: '<%= concat.dist.dest %>',
        dest: 'dist/<%= pkg.name %>.min.js'
      }
    },
    jshint: {
      options: {
        curly: true,
        eqeqeq: true,
        immed: true,
        latedef: true,
        newcap: true,
        noarg: true,
        sub: true,
        undef: true,
        unused: true,
        boss: true,
        eqnull: true,
        browser: true,
        globals: {}
      },
      gruntfile: {
        src: 'Gruntfile.js'
      },
      lib_test: {
        src: ['lib/**/*.js', 'test/**/*.js']
      }
    },
    qunit: {
      files: ['test/**/*.html']
    },
    watch: {
      gruntfile: {
        files: '<%= jshint.gruntfile.src %>',
        tasks: ['jshint:gruntfile']
      },
      lib_test: {
        files: '<%= jshint.lib_test.src %>',
        tasks: ['jshint:lib_test', 'qunit']
      }
    }
  });

  // These plugins provide necessary tasks.
  grunt.loadNpmTasks('grunt-contrib-concat');
  grunt.loadNpmTasks('grunt-contrib-uglify');
  grunt.loadNpmTasks('grunt-contrib-qunit');
  grunt.loadNpmTasks('grunt-contrib-jshint');
  grunt.loadNpmTasks('grunt-contrib-watch');

  // Default task.
  grunt.registerTask('default', ['jshint', 'qunit', 'concat', 'uglify']);

};

load-grunt-tasks plugin

In the original basic Gruntfile, we have to manually load our Grunt plugins, as

1
2
3

grunt.loadNpmTasks('grunt-contrib-concat');
grunt.loadNpmTasks('grunt-contrib-uglify');
grunt.loadNpmTasks('grunt-contrib-imagemin');

If you now uninstall the plugin via npm and update your package.json, but forget to update your Gruntfile, your build will break. With load-grunt-tasks plugin, you can collapse that down to the following one-liner:

1

require('load-grunt-tasks')(grunt);

After requiring the plugin, it will analyze your package.json file, determine which of the dependencies are Grunt plugins and load them all automatically.

load-grunt-config plugin

load-grunt-tasks shrunk your Gruntfile in code and complexity a little, but task configurations still remain in the Gruntfile (defined in grunt.initConfig). As you configure a large application, it will still become a very large file.

This is when load-grunt-config comes into play. load-grunt-config lets you break up your Gruntfile config by task. With load-grunt-config, your Gruntfile may look like this:

Gruntfile with load-grunt-config
1
2
3
4
5
6
7
8
9
10
11
12
13
14

module.exports = function(grunt) {

  var path = require('path');
  const appOptions = {
      data: {},
      configPath: [
          path.join(process.cwd(), '/grunt/tasks')
      ]
  };

  require('time-grunt')(grunt);
  require('load-grunt-config')(grunt, appOptions);

};

Note that load-grunt-config also includes load-grunt-tasks’s functionality. The task configurations live in files in folder ./grunt/tasks. By default, ./grunt folder is used but, in this example, using a custom path is demonstrated. In other words, our directory structure should be like this:

Directory structure
1
2
3
4
5
6

- current_project/
-- Gruntfile
-- grunt/tasks/
---- concat.js
---- uglify.js
---- imagemin.js

The task configuration for each task is defined in respective file name. For example, task concat is defined in “grunt/tasks/concat.js”:

grunt/tasks/concat.js
1
2
3
4
5
6
7
8
9
10

module.exports = {
  options: {
    banner: '<%= banner %>',
    stripBanners: true
  },
  dist: {
    src: ['lib/<%= pkg.name %>.js'],
    dest: 'dist/<%= pkg.name %>.js'
  }
};

The list of registered task aliases such as default is defined in aliases.js file.

grunt/tasks/aliases.js
1
2
3
4
5
6
7
8
9
10
11
12
13
14

module.exports = function(grunt, appOptions) {
    var buildList = [
        'jshint',
        'qunit',
        'concat',
        'uglify'
    ];

    return {
        default: ['build'],
        build: buildList,
        test: ['jslint']
    };
};

References

  • Safari: Introducing Grunt: the JavaScript task runner
  • Common Grunt plugins:
    • load-grunt-config: key plugin to keep Gruntfile organized.
    • concat
    • Unit Testing: qunit
    • Image optimization: imagemin
    • Deploying: deploy
    • Chaining: concurrent
  • Project scaffolding with grunt-init
    • grunt-init-commonjs - Create a commonjs module, including Nodeunit unit tests.
    • grunt-init-gruntfile - Create a basic Gruntfile.
    • grunt-init-gruntplugin - Create a Grunt plugin, including Nodeunit unit tests.
    • grunt-init-jquery - Create a jQuery plugin, including QUnit unit tests.
    • grunt-init-node - Create a Node.js module, including Nodeunit unit tests.
]]> <![CDATA[DSL Implementation in Groovy]]> 2017-08-13T15:18:30-07:00 http://tdongsi.github.io/blog/2017/08/13/groovy-dsl Domain-Specific Language is a mini language for a specific problem and/or in a narrow context. For example, internally used automation tools usually define some small DSL for configuration and most users understand the context and what DSL offers.

This blog post offers my simplistic view of how an internal DSL is implemented in Groovy via closure delegation. It shows the progression from standard Java-like implementation -> its fluent version -> final DSL form. This might help undrestanding the inner workings of a DSL such as Jenkins’s Pipeline steps. There are probably more advanced methods/frameworks for creating DSL. However, those are not in the scope of this post.

Example DSL

We want to implement a simple DSL that is similar to Pipeline steps in Jenkinsfile.

DSL in Jenkinsfile
1
2
3
4
5
6

{
    withEnv("PATH=/usr/bin")
    echo("Starting pipeline")
    sh("ls .")
    error("Error here")
}

In this DSL example, users will write a sequence of steps using a small, pre-defined set of custom statements such as echo and sh above. For each step in the DSL, the backend classes and objects will perform some execution in the background, using the relevant context specific to the domain (e.g., Jenkins domain). For simplicity, println statements will be used in the following examples.

The advantage of DSL is that the developers can implement the backend in some fully-featured language such as Java but the users don’t need to know such language to use it. Such a separation is common in DevOps and automation frameworks where the users want the flexibility of configuring based on their needs but don’t want to get exposed to the implementation details (which are usually ugly and compplicated). Instead, the users only need to learn the DSL to use it while still have the flexibility to do what they want. One example can be found in data science domain where data scientists are usually more comfortable developing in R or SQL but automated deployment frameworks or tools can be in another language such as Java.

Version 1: Java-like standard implementation

First, we show a standard implementation in Java to show how backend execution can be implemented. In the advanced versions, the difference is only in its public interface to make it more user-friendly but the backend execution will be similar.

Standard Java implementation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

/**
 * Java code with standard implementation
 * Try to simulate some kind of DSL like Pipeline steps in Jenkins
 */
class JavaDsl {

    void echo(String message) {
        println "Echo: $message";
    }

    void sh(String script) {
        println "Shell: $script";
    }

    void error(String message) {
        println "Error here: $message";
    }

    // A more advanced DSL
    void withEnv(String var) {
        println "Using: $var";
    }

    void execute() {
        println "Executing ...";
    }

}

println "1) Standard Java implementation";
JavaDsl javaDsl = new JavaDsl();
javaDsl.withEnv("PATH=/usr/bin");
javaDsl.echo("Starting pipeline");
javaDsl.sh("ls .");
javaDsl.error("Error here");
javaDsl.execute();
println "";

The problem of this approach is that users have to write Java (or Groovy) code directly to use it.

Version 2: Fluent interface with Builder pattern

Fluent Java implementation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

/**
 * Java code with Builder pattern
 * Try to simulate some kind of DSL like Pipeline steps in Jenkins
 */
class JavaBuilderDsl {

    JavaBuilderDsl echo(String message) {
        println "Echo: $message"
        return this
    }

    JavaBuilderDsl sh(String script) {
        println "Shell: $script"
        return this
    }

    JavaBuilderDsl error(String message) {
        println "Error here: $message"
        return this
    }

    // A more advanced DSL
    JavaBuilderDsl withEnv(String var) {
        println "Using: $var"
        return this
    }

    void execute() {
        println "Executing ..."
    }
}

println "2) Fluent Java implementation (Builder)"
JavaBuilderDsl builderDsl = new JavaBuilderDsl()
builderDsl.withEnv("PATH=/usr/bin")
        .echo("Starting pipeline")
        .sh("ls .")
        .error("Error here")
        .execute()
println ""

In this version, the Build design pattern is used in the implementation. As shown above, the code is much more fluent with the object name builderDsl not being repeated every single line. As a result, the code is less verbose and much more user-friendly.

Version 3: DSL with Groovy closure

Standard Groovy implementation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

/**
 * Groovy code with standard implementation
 * Try to simulate some kind of DSL like Pipeline steps in Jenkins
 */
class GroovyDsl {

    def echo(String message) {
        println "Echo: $message"
    }

    def sh(String script) {
        println "Shell: $script"
    }

    def error(String message) {
        println "Error here: $message"
    }

    // A more advanced DSL
    def withEnv(String var) {
        println "Using: $var"
    }

    static void execute(closure) {
        GroovyDsl body = new GroovyDsl()
        closure(body)
        println "Executing ..."
    }

}

println "3) Standard Groovy implementation"
GroovyDsl.execute { dsl ->
    dsl.withEnv("PATH=/usr/bin")
    dsl.echo("Starting pipeline")
    dsl.sh("ls .")
    dsl.error("Error here")
}
println ""

This first version of Groovy implementation is presented here to show connection with its Java counterparts. As shown below, the input variable dsl in the closure can be abstracted away using delegate.

Transparent DSL with delegate
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

class GroovyDsl {

    def echo(String message) {
        println "Echo: $message"
    }

    def sh(String script) {
        println "Shell: $script"
    }

    def error(String message) {
        println "Error here: $message"
    }

    // A more advanced DSL
    def withEnv(String var) {
        println "Using: $var"
    }

    static void execute(Closure closure) {
        GroovyDsl body = new GroovyDsl()
        // TRICKY: Modify the input var? Hmmm.
        closure.delegate = body
        closure()
        println "Executing ..."
    }

    static void executeBest(Closure closure) {
        GroovyDsl body = new GroovyDsl()
        body.with(closure)
        println "Executing ..."
    }

}

println "4) DSL-style Groovy implementation"
GroovyDsl.execute {
    withEnv("PATH=/usr/bin")
    echo("Starting pipeline")
    sh("ls .")
    error("Error here")
}
println ""

println "4b) DSL-style Groovy (better) implementation"
GroovyDsl.executeBest {
    withEnv("PATH=/usr/bin")
    echo("Starting pipeline")
    sh("ls .")
    error("Error here")
}
println ""

In this final version, only a very small boiler-plate code GroovyDsl.executeBest remains. The following lines form a mini language (i.e., DSL) that can be exposed to users. The users can start using the DSL without having to learn Groovy or Java.

Note that the executeBest is the equivalent but less straight-forward way to do the same thing with delegate. Compared with execute, it has the benefit of NOT modifying the input reference closure.

Reference

]]> <![CDATA[Advanced Jenkinsfile Cookbook]]> 2017-07-19T14:23:01-07:00 http://tdongsi.github.io/blog/2017/07/19/jenkinsfile-cookbook This post details some of the more advanced Jenkins pipelines using Jenkinsfile.

Nexus authentication in Maven

More detailed discussion is in here.

Jenkinsfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

def myScript

pipeline {
   agent { node { label 'test-agent' } }
   stages {
       stage("compile") {
           steps {
               checkout scm
               withCredentials([
                 [$class: 'StringBinding', credentialsId: 'nexusUsername', variable: 'nexusUsername'],
                 [$class: 'StringBinding', credentialsId: 'nexusPassword', variable: 'nexusPassword']
               ]) {
                   script {
                       myScript = load 'jenkins/xml.groovy'
                       def xmlTemplate = readFile( 'jenkins/settings.xml' )
                       String xmlFile = myScript.transformXml(xmlTemplate, env.nexusUsername, env.nexusPassword)

                       String myPath = 'temp.xml'
                       writeFile file: myPath, text: xmlFile

                       sh "mvn -B clean compile -s ${myPath}"

                       sh "rm ${myPath}"
                   }
               }
           }
           post {
           failure {
               echo "Sending email for compile failed (TBD)"
            }
           }
       }
   }
}
xml.groovy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import groovy.xml.StreamingMarkupBuilder
import groovy.xml.XmlUtil

@NonCPS
def transformXml(String xmlContent, String username, String password) {
  def xml = new XmlSlurper(false, false).parseText(xmlContent)

  echo 'Start tranforming XML'
  xml.servers.server.each { node ->
    node.username = username
    node.password = password
  }

  def outWriter = new StringWriter()
  XmlUtil.serialize( xml, outWriter )
  return outWriter.toString()
}

return this

Running test suite in parallel

It is possible to take a Maven/JUnit-based test suite that takes too long to run on a single node and parallelize the test execution across multiple nodes instead. The Parallel Test Executor Plugin is exactly for that purpose.

Jenkinsfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

node('remote') {
  git 'https://github.com/jenkinsci/parallel-test-executor-plugin-sample.git'
  stash name: 'sources', includes: 'pom.xml,src/'
}
def splits = splitTests count(2)
def branches = [:]
for (int i = 0; i < splits.size(); i++) {
  def index = i // fresh variable per iteration; i will be mutated
  branches["split${i}"] = {
    node('remote') {
      deleteDir()
      unstash 'sources'
      def exclusions = splits.get(index);
      writeFile file: 'exclusions.txt', text: exclusions.join("\n")
      sh "${tool 'M3'}/bin/mvn -B -Dmaven.test.failure.ignore test"
      junit 'target/surefire-reports/*.xml'
    }
  }
}
parallel branches

Note that, this is different from modiyfing test harness (e.g., JUnit, TestNG) to parallelize the test execution on a single node. It could be time-consuming and risk destabalizing the tests while the chance of success is usually small.

More details can be found in the following links:

]]> <![CDATA[Basic Jenkinsfile Cookbook]]> 2017-07-18T11:20:20-07:00 http://tdongsi.github.io/blog/2017/07/18/basic-jenkinsfile-cookbook This post shows how to customize standard Pipeline “steps” in Jenkinsfile besides their common usage.

List of basic Jenkinsfile steps in this post:

  • checkout/git
  • emailext
  • findFiles
  • input
  • junit
  • parameters/properties
  • podTemplates
  • sendSlack
  • stash/unstash
  • withCredentials

checkout/git step

scm is the global variable for the current commit AND branch AND repository of Jenkinsfile. checkout scm means checking out all other files with same version as the Jenkinsfile associated with running pipeline. To check out another repository, you need to specify the paremeters to checkout step.

Checkout from another Git repo
1
2
3
4
5
6
7
8
9
10

checkout([$class: 'GitSCM', branches: [[name: '*/master']],
     userRemoteConfigs: [[url: 'http://git-server/user/repository.git']]])

// From README file.
checkout scm: [$class: 'MercurialSCM', source: 'ssh://hg@bitbucket.org/user/repo', clean: true, credentialsId: '1234-5678-abcd'], poll: false
// If scm is the only parameter, you can omit its name, but Groovy syntax then requires parentheses around the value:
checkout([$class: 'MercurialSCM', source: 'ssh://hg@bitbucket.org/user/repo'])

// Short hand form for Git
git branch: 'develop', url: 'https://github.com/WtfJoke/Any.git'

Reference:

emailext step

To send email as HTML page, set content type to HTML and use content as ${FILE,path="email.html"}. In Jenkinsfile, the code should look like this:

Send HTML report as email
1
2
3
4
5
6
7
8

emailext(
    subject: 'Deploy Notice',
    to: EMAIL_AUDIENCE,
    body: '${FILE,path="deploy_email.html"}',
    presendScript: '$DEFAULT_PRESEND_SCRIPT',
    replyTo: 'devops@my.company.com',
    mimeType: 'text/html'   // email as HTML
)

Note that it’s single-quoted strings, not double-quoted, being used for body and presendScript parameters in the example code above.

Reference:

findFiles step

Doing in Bash:

Doing in Bash
1
2
3
4
5
6
7
8

sh '''
for file in target/surefire-reports/*.txt;
do
    echo $file >> testresult
done
cat testresult
'''
def result = readFile "testresult"
Doing in Groovy
1
2
3
4
5
6

def files = findFiles(glob: 'target/surefire-reports/*.txt')
for file in files:
  echo """
  ${files[0].name} ${files[0].path} ${files[0].directory} 
  ${files[0].length} ${files[0].lastModified}
  """

Reference:

input step

Simple input step can be used to ask for approval to proceed. For asking input from a list of multiple choices, you can use the advanced version of input.

Input from list of choices
1
2
3
4
5
6

   sh "source scripts/findCL.sh > choiceLists.txt"
   def choiceOptions = readFile "${env.WORKSPACE}/choiceLists.txt"
   def choice = input(
   id: 'CHOICE_LIST', message:'Choose a CL' , parameters: [
    [$class: 'ChoiceParameterDefinition', name:'CHOICE_LIST_SELECTED', description:'Select one', choices:choiceOptions]
  ])

Reference:

junit step

JUnit tests + PMD, FindBugs, CheckStyle. In Blue Ocean interface, these will be displayed in a separate tab.

Related steps
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

stage('JUnit-Reports'){
    junit allowEmptyResults: true, testResults: '**/build/test-results/*.xml'
}

stage('FindBugs-Reports'){
    step([$class: 'FindBugsPublisher', canComputeNew: false, defaultEncoding: '',
    excludePattern: '', healthy: '', includePattern: '',
    pattern: '**/build/reports/findbugs/*.xml', unHealthy: ''])
}

stage('PMD-Reports'){
    step([$class: 'PmdPublisher', canComputeNew: false, defaultEncoding: '',
    healthy: '', pattern: '**/build/reports/pmd/*.xml', unHealthy: ''])
}

stage('CheckStyle-Reports'){
    step([$class: 'CheckStylePublisher', canComputeNew: false, defaultEncoding: '',
    healthy: '', pattern: '**/build/reports/checkstyle/*.xml', unHealthy: ''])
}

parameters/properties step

parameters step adds certain job parameters for the overall pipeline job.

parameters step in Declarative pipeline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

pipeline {
    options {
        buildDiscarder(logRotator(numToKeepStr: '30', artifactNumToKeepStr: '30'))
        disableConcurrentBuilds()
    }
    agent { node { label 'aqueduct-agent' } }
    parameters {
        choice(name: 'ClusterName', choices: 'func\ninteg\nperf', description: 'Name of the cluster to test.')
    }
    stages {
        stage("Build") {
            steps {
                echo "Hello"
                ...
            }
        } //  end of stage
    }
    post {
        always {
            ...
        }
    }
}

In Scripted pipeline, its equivalent counterpart is properties step, as shown below.

parameters step for Scripted pipeline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

properties(
    [
        [
            $class  : 'jenkins.model.BuildDiscarderProperty',
            strategy: [
                $class      : 'LogRotator',
                numToKeepStr: '20'
            ]
        ],
        pipelineTriggers(
            [
                [
                    $class: 'hudson.triggers.TimerTrigger',
                    spec  : "H 8 * * *"
                ]
            ]
        )
    ]
)

node('agent') {
    stage('Checkout') {
        checkout scm
    }
    ...
}

In the Jenkins UI, this will be converted to configurations when you click on “View Configuration” for that job, as shown in screenshot below. Note that the configurations in this page is read-only when using Jenkinsfile. Any modifications made to the page will be ignored, leaving configurations set in Jenkinsfile final (“Infrastructure as Code”).

Screenshot of View Configuration page

Reference:

podTemplate step

This step is used to specify a new pod template for running jobs on Kubernetes cluster.

Kubernetes plugin
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

podTemplate(label:'base-agent', containers: [
    containerTemplate(name: 'maven',
        image: 'docker.my/tdongsi/jenkins-agent:13',
        workingDir: '/home/jenkins',
        volumes: [hostPathVolume(mountPath: '/srv/jenkins', hostPath: '/usr/local/npm'),
        secretVolume(mountPath: '/etc/mount2', secretName: 'my-secret')],
        imagePullSecrets: 'sfregistry')
]) {
    node('base-agent') {
        stage('Checkout') {
            checkout scm
        }

        stage('main') {
            sh 'java -version'
            sh 'mvn -version'
            sh 'python -V'
        }

        input 'Finished with K8S pod?'
    }
}

Reference:

sendSlack step

Standard Jenkinsfile for testing Slack

Jenkinsfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

node('test-agent') {
    stage('Checkout') {
        checkout scm
    }

    stage('Main') {
        withCredentials([string(credentialsId: 'matrixsfdc-slack', variable: 'TOKEN')]) {
            slackSend ( teamDomain: 'matrixsfdc', channel: '#jenkins-pcloud', token: env.TOKEN,
                   baseUrl: 'https://matrixsfdc.slack.com/services/hooks/jenkins-ci/',
                   color: '#FFFF00',
                   message: "STARTED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]' (${env.BUILD_URL})"
                   )
        }
    }

    input 'Finished with K8S pod?'
}

stash/unstash steps

stash step can be used to save a set of files, to be unstashed later in the same build, generally for using in another workspace. unstash will restore the files into the same relative locations as when they are stashed. If you want to change the base directory of the stashed files, you should wrap the stash steps in dir step.

We should use stash/unstash to avoid the common anti-pattern of copying files into some special, globally visible directory such as Jenkins home or one of its subdirectories. Using such anti-pattern will make it hard to support many jobs for many users since, eventually, there will be some name clash and, subsequently, some convoluted naming of those files to avoid such name clashes.

Note that stash and unstash steps are designed for use with small files. If the size is above 5 MB, we should consider an alternative such as Nexus/Artifactory for jar files, blob stores for images.

Example usage of stash and unstash:

stash/unstash example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

    stage('Stash') {
        dir ('test') {
            sh '''
                cd release_notes
                touch deployment_ids.json
                touch deployment_summary.json
                ls -al
            '''

            stash name: 'deployment_ids', includes: 'release_notes/deployment_ids.json'
            stash name: 'deployment_summary', includes: 'release_notes/deployment_summary.json'

        } // end dir
    } // end stage 'Stash'

    stage('Check unstash') {
        dir('check') {
            unstash 'deployment_ids'

            sh 'tree -L 2'
        }

        dir('check2') {
            unstash 'deployment_summary'

            sh 'tree -L 2'
        }
    }

    stage('Clean up') {
        sh 'tree -L 2'
        deleteDir()
    }

Example output:

Console output of the above stash/unstash example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

[Pipeline] sh
04:14:04 + cd release_notes
04:14:04 + touch deployment_ids.json
04:14:04 + touch deployment_summary.json
04:14:04 + ls -al
04:14:04 total 128
04:14:04 drwxr-xr-x  5 jenkins cdrom  4096 Aug 18 04:14 .
04:14:04 drwxr-xr-x 26 jenkins cdrom  4096 Aug 18 04:14 ..
04:14:04 -rw-r--r--  1 jenkins cdrom     0 Aug 18 04:14 deployment_ids.json
04:14:04 -rw-r--r--  1 jenkins cdrom     0 Aug 18 04:14 deployment_summary.json
[Pipeline] stash
04:14:04 Stashed 1 file(s)
[Pipeline] stash
04:14:04 Stashed 1 file(s)
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Check unstash)
[Pipeline] dir
04:14:04 Running in /var/lib/jenkins/workspace/feature_test-23NL25SRLW3W6WXNVW2NXBADUXIYBTCBZAO5YRKVQPVT3NUSEOTQ/check
[Pipeline] {
[Pipeline] unstash
[Pipeline] sh
04:14:04 [check] Running shell script
04:14:05 + tree -L 2
04:14:05 .
04:14:05 └── release_notes
04:14:05     └── deployment_ids.json
04:14:05 
04:14:05 1 directory, 1 file
[Pipeline] }
[Pipeline] // dir
[Pipeline] dir
04:14:05 Running in /var/lib/jenkins/workspace/feature_test-23NL25SRLW3W6WXNVW2NXBADUXIYBTCBZAO5YRKVQPVT3NUSEOTQ/check2
[Pipeline] {
[Pipeline] unstash
[Pipeline] sh
04:14:05 [check2] Running shell script
04:14:05 + tree -L 2
04:14:05 .
04:14:05 └── release_notes
04:14:05     └── deployment_summary.json
04:14:05 
04:14:05 1 directory, 1 file
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Clean up)
[Pipeline] sh
04:14:05 [feature_test-23NL25SRLW3W6WXNVW2NXBADUXIYBTCBZAO5YRKVQPVT3NUSEOTQ] Running shell script
04:14:05 + tree -L 2
04:14:05 .
04:14:05 ├── check
04:14:05 │   └── release_notes
04:14:05 ├── check2
04:14:05 │   └── release_notes
04:14:05 ├── check2@tmp
04:14:05 ├── check@tmp
04:14:05 ├── Jenkinsfile
...

withCredentials step

There are different variations of withCredentials step. The most common ones are:

Binding secret to username and password separately
1
2
3
4
5
6
7
8

node {
    withCredentials([usernamePassword(credentialsId: 'amazon', usernameVariable: 'USERNAME', passwordVariable: 'PASSWORD')]) {
        // available as an env variable, but will be masked if you try to print it out any which way
        sh 'echo $PASSWORD'
        // also available as a Groovy variable—note double quotes for string interpolation
        echo "$USERNAME"
    }
}
Binding secret to $username:$password
1
2
3
4
5
6
7
8

node {
  withCredentials([usernameColonPassword(credentialsId: 'mylogin', variable: 'USERPASS')]) {
    sh '''
      set +x
      curl -u $USERPASS https://private.server/ > output
    '''
  }
}
Binding secret string to a variable
1
2
3
4
5
6
7

node {
  withCredentials([string(credentialsId: 'secretString', variable: 'MY_STRING')]) {
    sh '''
      echo $MY_STRING
    '''
  }
}

For secret file, the file will be passed into some secret location and that secret location will be bound to some variable. If you want the secret files in specific locations, the workaround is to create symlinks to those secret files.

Binding secret file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

    withCredentials( [file(credentialsId: 'host-cert', variable: 'HOST_CERT'),
                    file(credentialsId: 'host-key', variable: 'HOST_KEY'),
                    file(credentialsId: 'cert-ca', variable: 'CERT_CA')
                    ])
    {
        sh """
            mkdir download
            ln -s ${env.HOST_CERT} download/hostcert.crt
            ln -s ${env.HOST_KEY} download/hostcert.key
            ln -s ${env.CERT_CA} download/ca.crt
        """

        // The Python script read those files download/*.* by default
        sh "python python/main.py"
    }

For “private key with passphrase” Credential type, sshagent is only usage that I know (credential ID is jenkins_ssh_key in this example):

Binding private key with passphrase
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

node('my-agent'){
  stage 'Checkout'
  checkout scm

  if (env.BRANCH_NAME == 'master') {
    stage 'Commit'
    println "Pushing Jenkins Shared Library"

    sshagent(['jenkins_ssh_key']) {
      sh """
        git branch master
        git checkout master
        ssh-keyscan -H -p 12222 \${JENKINS_ADDR} >> ~/.ssh/known_hosts
        git remote add jenkins ssh://tdongsi@\${JENKINS_ADDR}:12222/workflowLibs.git
        git push --force jenkins master
      """
    }

  }
}

Reference:

References

]]> <![CDATA[Troubleshooting Groovy Code in Jenkinsfile]]> 2017-06-16T23:52:44-07:00 http://tdongsi.github.io/blog/2017/06/16/troubleshooting-groovy-scripts-in-jenkinsfile In this post, we look into some troubleshooting tips when using independent Groovy scripts in Jenkins pipeline and how to work around those.

Cannot load a Groovy script in Declarative Pipeline

Problem: Loading Groovy methods from a file with load step does not work inside Declarative Pipeline step, as reported in this issue.

Workaround: There are a few work-arounds. The most straight-forward one is to use script step.

Loading Groovy script
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

steps {
    checkout scm
    withCredentials([
        [$class: 'StringBinding', credentialsId: 'nexusUserName', variable: 'nexusUserName'],
        [$class: 'StringBinding', credentialsId: 'nexusPassword', variable: 'nexusPassword']
    ]) {
        script {
            myScript = load 'jenkins/xml.groovy'
            String myFile = myScript.transformXml(settingsFile, env.nexusUserName, env.nexusPassword)
            sh "mvn -B -s ${myFile} clean compile"

            sh "rm ${myFile}"
        }
    }
}

You can also define Groovy methods from inside the Jenkinsfile.

Example Jenkinsfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

import groovy.xml.StreamingMarkupBuilder
import groovy.xml.XmlUtil

@NonCPS
def xmlTransform(txt, username, password) {

    def xmlRoot = new XmlSlurper(false, false).parseText(txt)
    echo 'Start tranforming XML'
    xmlRoot.servers.server.each { node ->
       node.username = username
       node.password = password
    }

    // TRICKY: FileWriter does NOT work
    def outWriter = new StringWriter()
    XmlUtil.serialize( xmlRoot, outWriter )
    return outWriter.toString()
}

...

   steps {
        checkout scm
        withCredentials([
            [$class: 'StringBinding', credentialsId: 'nexusUserName', variable: 'nexusUserName'],
            [$class: 'StringBinding', credentialsId: 'nexusPassword', variable: 'nexusPassword']
        ]) {
            script {
                myScript = load 'jenkins/xml.groovy'
                String myFile = xmlTransform(settingsFile, env.nexusUserName, env.nexusPassword)
                sh "mvn -B -s ${myFile} clean compile"

                sh "rm ${myFile}"
            }
        }
    }

For Declarative Pipeline, to reuse the code from a Groovy script, you must use Shared Libraries. Shared Libraries are not specific to Declarative; they were released some time ago and were seen in Scripted Pipeline. This blog post discusses an older mechanism for Shared Library. For the newer mechanism of importing library, please check out this blog post. Due to Declarative Pipeline’s lack of support for defining methods, Shared Libraries definitely take on a vital role for code-reuse in Jenkinsfile.

File reading and writing not supported

Java/Grooy reading and writing using “java.io.File” class is not directly supported.

Using File class does NOT work
1

def myFile = new File('/home/data/myfile.xml')

In fact, using that class in Jenkinsfile must go through “In-Process Script Approval” with this warning.

new java.io.File java.lang.String Approving this signature may introduce a security vulnerability! You are advised to deny it.

Even then, “java.io.File” will refer to files on the master (where Jenkins is running), not the current workspace on Jenkins slave (or slave container). As a result, it will report the following error even though the file is present in filesystem (relevant Stackoverflow) on slave:

1
2

java.io.FileNotFoundException: /home/data/myfile.xml (No such file or directory)
  at java.io.FileInputStream.open0(Native Method)

That also means related class such as FileWriter will NOT work as expected. It reports no error during execution but you will find no file since those files are created on Jenkins master.

Workaround:

  • For file reading, use readFile step.
  • For file writing, use writeFile step. However, Pipeline steps (such as writeFile) are NOT allowed in @NonCPS methods. For more complex file writing, you might want to export the file content as String and use the following code snippet:
Shell command
1
2
3
4
5
6

String xmlFile = ...

// TRICKY: FileWriter does NOT work in xmlTransform
def mCommand = "cat >${settingsFile} <<EOF"
mCommand += "\n${xmlFile}\nEOF"
sh mCommand

In the code snippet above, we construct a here document-formatted command for writing multi-line string in mCommand before passing to sh step for executing.

heredoc example to explain mCommand
1
2
3
4
5
6
7
8
9
10

$ cat >output.txt <<EOF
SELECT foo, bar FROM db
WHERE foo='baz'
More line from xmlFile
EOF

$ cat output.txt
SELECT foo, bar FROM db
WHERE foo='baz'
More line from xmlFile

Serialization errors

You often encounter this type of errors when using non-serialiable classes from Groovy/Java libraries.

Error in Jenkins log
1
2

java.io.NotSerializableException: org.codehaus.groovy.control.ErrorCollector
  at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:860)
Related error in Jenkins log
1
2
3
4
5
6

java.lang.UnsupportedOperationException: Calling public static java.lang.Iterable 
org.codehaus.groovy.runtime.DefaultGroovyMethods.each(java.lang.Iterable,groovy.lang.Closure) on a
CPS-transformed closure is not yet supported (JENKINS-26481); 
encapsulate in a @NonCPS method, or use Java-style loops
  at org.jenkinsci.plugins.workflow.cps.GroovyClassLoaderWhitelist.checkJenkins26481
    (GroovyClassLoaderWhitelist.java:90)

There is also some known issue about JsonSlurper. These problems come from the fact that variables in Jenkins pipelines must be serializable. Since pipeline must survive a Jenkins restart, the state of the running program is periodically saved to disk for possible resume later. Any “live” objects such as a network connection is not serializble.

Workaround: Explicitly discard non-serializable objects or use @NonCPS methods.

Quoted from here: @NonCPS methods may safely use non-Serializable objects as local variables, though they should NOT accept nonserializable parameters or return or store nonserializable values. You may NOT call regular (CPS-transformed) methods, or Pipeline steps, from a @NonCPS method, so they are best used for performing some calculations before passing a summary back to the main script.

References

]]> <![CDATA[Groovy in Jenkinsfile]]> 2017-06-07T12:08:15-07:00 http://tdongsi.github.io/blog/2017/06/07/groovy-in-jenkinsfile Groovy is supported in Jenkinsfile for quick scripting. However, lots of features in the Groovy language is not supported and simple works in Groovy can be really tricky in Jenkinsfile.

Different ways to process XML file

In summary, if it is possible, use another script language (e.g., Python) for file manipulation in Jenkinsfile. It is time consuming to navigate all tricky stuffs of Groovy implementaiton in Jenkins:

  • In-process Script Approval: you have to approve every single class and method one by one.
  • Some features of Groovy is not supported and it takes time to figure out what is not supported and how to work around. When in doubt, use @NonCPS.

Groovy method in Jenkinsfile

Jenkinsfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

import groovy.xml.StreamingMarkupBuilder
import groovy.xml.XmlUtil

def settingsFile = 'temp.xml'

@NonCPS
def xmlTransform(txt, username, password) {

    def xmlRoot = new XmlSlurper(false, false).parseText(txt)
    echo 'Start tranforming XML'
    xmlRoot.servers.server.each { node ->
       node.username = username
       node.password = password
    }

    // TRICKY: FileWriter does NOT work
    def outWriter = new StringWriter()
    XmlUtil.serialize( xmlRoot, outWriter )
    return outWriter.toString()
}

pipeline {
   agent { node { label 'test-agent' } }
   stages {
       stage("compile") {
           steps {
               checkout scm
               withCredentials([
                 [$class: 'StringBinding', credentialsId: 'nexusUsername', variable: 'nexusUsername'],
                 [$class: 'StringBinding', credentialsId: 'nexusPassword', variable: 'nexusPassword']
               ]) {
                   script {
                       def xmlTemplate = readFile( 'jenkins/settings.xml' )
                       def xmlFile = xmlTransform(xmlTemplate, env.nexusUsername, env.nexusPassword)
                       writeFile file: settingsFile, text: xmlFile

                       sh "mvn -B -s ${settingsFile} clean compile"
                   }
               }
           }
           post {
           failure {
               echo "Sending email for compile failed (TBD)"
            }
           }
       }
   }
}

Some notes:

  • import statements must be at the top, right after the shebang and before anything else.
  • The Groovy methods must be annotated with @NonCPS or Jenkins will report the error “java.io.NotSerializableException”.
  • The Groovy methods can not be defined inside a step block. It must be defined at the top.
  • @NonCPS is required since the Groovy method uses several non-serializble objects.

Groovy method in separate script

Jenkinsfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

def myScript

pipeline {
   agent { node { label 'test-agent' } }
   stages {
       stage("compile") {
           steps {
               checkout scm
               withCredentials([
                 [$class: 'StringBinding', credentialsId: 'nexusUsername', variable: 'nexusUsername'],
                 [$class: 'StringBinding', credentialsId: 'nexusPassword', variable: 'nexusPassword']
               ]) {
                   script {
                       myScript = load 'jenkins/xml.groovy'
                       def xmlTemplate = readFile( 'jenkins/settings.xml' )
                       String xmlFile = myScript.transformXml(xmlTemplate, env.nexusUsername, env.nexusPassword)

                       String myPath = 'temp.xml'
                       def mCommand = "cat >${myPath} <<EOF"
                       mCommand += "\n${xmlFile}\nEOF"
                       sh mCommand

                       sh "mvn -B clean compile -s ${myPath}"

                       sh "rm ${myPath}"
                   }
               }
           }
           post {
           failure {
               echo "Sending email for compile failed (TBD)"
            }
           }
       }
   }
}
xml.groovy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import groovy.xml.StreamingMarkupBuilder
import groovy.xml.XmlUtil

@NonCPS
def transformXml(String xmlContent, String username, String password) {
  def xml = new XmlSlurper(false, false).parseText(xmlContent)

  echo 'Start tranforming XML'
  xml.servers.server.each { node ->
    node.username = username
    node.password = password
  }

  def outWriter = new StringWriter()
  XmlUtil.serialize( xml, outWriter )
  return outWriter.toString()
}

return this

Groovy method in shared library

The above Nexus authentication code is likely repeated across multiple Maven builds. Therefore, it is worth converting it into a DSL into a Shared Library in Jenkins. The DSL takes two parameters:

  • templateFile: settings.xml template with Nexus credentials info redacted.
  • command: Maven command with settings file NOT specified (i.e., NO “-s” option in the command).

The example usage is as follows:

Jenkinsfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

pipeline {
   agent { node { label 'test-agent' } }
   stages {
       stage("compile") {
           steps {
               checkout scm
               script {
                    withNexusMaven {
                        templateFile = 'jenkins/settings.xml'
                        command = "mvn -B clean compile"
                    }
               }
           }
           post {
           failure {
               echo "Sending email for compile failed (TBD)"
            }
           }
       }
   }
}

The Jenksinfile is much cleaner since most of implementation details have been moved inside the DSL:

withNexusMaven.groovy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

#!groovy
import groovy.xml.XmlUtil

@NonCPS
def transformXml(String xmlContent, String username, String password) {
  def xml = new XmlSlurper(false, false).parseText(xmlContent)

  echo 'Start tranforming XML'
  xml.servers.server.each { node ->
    node.username = username
    node.password = password
  }

  def outWriter = new StringWriter()
  XmlUtil.serialize( xml, outWriter )
  return outWriter.toString()
}

def call(Closure body) {

    def config = [:]
    if (body != null) {
        body.resolveStrategy = Closure.DELEGATE_FIRST
        body.delegate = config
        body()
    }

    def templateFile = config.templateFile ?: '/home/data/settings.xml'
    def command = config.command ?: "mvn -B clean compile"

    withCredentials([
        [$class: 'StringBinding', credentialsId: 'nexusUsername', variable: 'nexusUsername'],
        [$class: 'StringBinding', credentialsId: 'nexusPassword', variable: 'nexusPassword']
    ]) {
        def xmlTemplate = readFile templateFile
        String xmlFile = transformXml(xmlTemplate, env.nexusUsername, env.nexusPassword)

        String tempFile = 'temp.xml'
        writeFile file: tempFile, text: xmlFile

        sh "${command} -s ${tempFile}"

        // Clean up
        sh "rm ${tempFile}"
    }
}
]]> <![CDATA[Sending Emails From Docker Containers]]> 2017-05-25T13:42:45-07:00 http://tdongsi.github.io/blog/2017/05/25/sending-emails-from-docker-containers In this post, we looks into how to set up notification emails at the end of CI pipelines in a containerized Jenkins system. First, we look into conventional Jenkins system (directly hosted) that has direct communication to the SMTP server. After that, we will look into adjustments required for a containerized Jenkins system to run in the same environment.

Sending emails in standard Jenkins setup

We first look at a typical Jenkins setup, where the Jenkins instance is installed directly on a host machine (VM or bare-metal) and has direct communication to the SMTP server. For corporate network, you may have to use an SMTP relay server instead. For those cases, you can configure SMTP communication by setting up Postfix. In CentOS, it could be a simple “sudo yum install -y mailx”.

After installing, update /etc/postfix/main.cf with correct relay information: myhostname, myorigin, mydestination, relayhost, alias_maps, alias_database. An example /etc/postfix/main.cf is shown below:

/etc/postfix/main.cf example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

# See /usr/share/postfix/main.cf.bak for a commented, more complete version

myhostname = dev-worker-1.example.com
smtpd_banner = $myhostname ESMTP $mail_name
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.


alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = dev-worker-1.example.com
mydestination = dev-worker-1.example.com, localhost.example.com, localhost
relayhost = smtprelay-prd.example.com
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = localhost
inet_protocols = all

We can test the setup by sending a test email with the following command:

Send a test email
1
2
3

[tdongsi@dev-worker-1 ~]# echo "Test localhost" | mailx -s Test tdongsi@example.com
send-mail: warning: inet_protocols: disabling IPv6 name/address support: Address family not supported by protocol
postdrop: warning: inet_protocols: disabling IPv6 name/address support: Address family not supported by protocol

After the postfix service is up, Jenkins can be configured to send email with Mailer plugin. Mail server can be configured in Manage Jenkins page, E-mail Notification section. Please visit Kohei Nozaki’s blog post for more detailed instructions and screenshots. We can also test the configuration by sending test e-mail in the same E-mail Notification section.

Sending email from container

Many Jenkins-based CI systems have been containerized and deployed on Kubernetes cluster (in conjunction with Kubernetes plugin). For email notifications in such CI systems, one option is to reuse postfix service, which is usually configured and ready on the Kubernetes nodes, and expose it to the Docker containers.

There are two changes need to be made on Postfix to expose it to Docker containers on one host.

  1. Exposing Postfix to the docker network, that is, Postfix must be configured to bind to localhost as well as the docker network.
  2. Accepting all incoming connections which come from any Docker containers.

Docker bridge (docker0) acts a a bridge between your ethernet port and docker containers so that data can go back and forth. We achieve the first requirement by adding the IP of docker0 to inet_iterfaces.

ifconfig example output
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

[centos@dev-worker-1 ~]$ ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1472
        inet 172.22.91.1  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 02:42:88:5f:24:28  txqueuelen 0  (Ethernet)
        RX packets 8624183  bytes 18891507332 (17.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15891332  bytes 16911210191 (15.7 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1472
        inet 172.22.91.0  netmask 255.255.0.0  destination 172.22.91.0
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 10508237  bytes 7051646109 (6.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 15511583  bytes 18744591891 (17.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

For the second requirement, the whole docker network as well as localhost should be added to mynetworks. In our kubernetes setup, the docker network should be flannel0 and its subnet’s CIDR notation is added to the mynetworks line:

Modified "/etc/postfix/main.cf"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# See /usr/share/postfix/main.cf.bak for a commented, more complete version

myhostname = dev-worker-1.example.com
smtpd_banner = $myhostname ESMTP $mail_name
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = dev-worker-1.example.com
mydestination = dev-worker-1.example.com, localhost.example.com, localhost
relayhost = smtprelay-prd.example.com
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 172.22.0.0/16
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = localhost, 172.22.91.1
inet_protocols = all

Note the differences in inet_interfaces and mynetworks from the last section. One can simply enter the Docker container/Kubernetes pod to verify such setup. Note that application mailx maybe not available in a container since we tend to keep the containers light-weight. Instead, prepare a sendmail.txt file (based on this) with the following SMTP commands and use nc to send out the email as shown below.

Send test email from container
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

mymac:k8s tdongsi$ kubectl --kubeconfig kubeconfig --namespace jenkins exec -it jenkins-8hgsn -- bash -il

jenkins@jenkins-8hgsn:~/test$ cat sendmail.txt
HELO x
MAIL FROM: test@example.com
RCPT TO: tdongsi@example.com
DATA
From: test@example.com
To: $YOUR_EMAIL
Subject: This is a test

The test is successful

.
quit

jenkins@jenkins-8hgsn:~/test$ nc 172.22.91.1 25 <sendmail.txt
220 dev-worker-1.eng.sfdc.net ESMTP Postfix
250 dev-worker-1.eng.sfdc.net
250 2.1.0 Ok
250 2.1.5 Ok
354 End data with <CR><LF>.<CR><LF>
250 2.0.0 Ok: queued as 1EF9E60C34
221 2.0.0 Bye

For containerized Jenkins system, mail server can also be configured in same Manage Jenkins page, E-mail Notification section. The only difference is the IP/hostname provided to SMTP server option. Instead of providing the known SMTP server’s IP and host, one should use the IP of docker0, as explained above. In the case of many nodes in Kubernetes cluster with different docker0 IP, the Docker container of Jenkins master should reside only on one host and docker0’s IP on that host should be used.

References

postfix version used in this post
1
2

[tdongsi@dev-worker-1 ~]$ postconf -v | grep mail_version
mail_version = 2.10.1
]]> <![CDATA[Maven and Gradle Builds in Jenkinsfile]]> 2017-05-20T23:27:08-07:00 http://tdongsi.github.io/blog/2017/05/20/gradle-settings-in-jenkinsfile In this post, we will look into how to securely authenticate with Nexus for Maven and Gradle builds in Jenkins pipelines. Nexus username and password should NOT be stored in plain text on Jenkins slaves or Docker images. Instead, those credentials should be passed into Jenkins pipeline using withCredentials step.

Maven

Maven builds in corporates usually use private repositories on Nexus, instead of public ones in Maven Central Repository. To do that, we usually configure Maven to check Nexus instead of the default, built-in connection to Maven Central. These configurations is stored in ~/.m2/settings.xml file.

For authentication with Nexus and for deployment, we must provide credentials accordingly. We usually add the credentials into our Maven Settings in settings.xml file.

Example Credentials in settings.xml
1
2
3
4
5
6
7
8
9

<settings>
  <servers>
    <server>
      <id>nexus</id>
      <username>deployment</username>
      <password>deployment123</password>
    </server>
  </servers>
</settings>

However, for automated build and deployment in Jenkins pipelines, it is not safe to store credentials in plain text files. Instead, one should store Nexus credentials as secrets in Jenkins and pass them into Jenkinsfile using their IDs (credentialsId). See this article for the full picture of related plugins used for storing and passing secrets in Jenkins.

Nexus authentication for Maven in Jenkinsfile.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

  withCredentials([
    [$class: 'StringBinding', credentialsId: 'nexusUsername', variable: 'nexusUsername'],
    [$class: 'StringBinding', credentialsId: 'nexusPassword', variable: 'nexusPassword']
  ]) {
    withEnv([
      'nexusPublic=https://nexus.example.com/nexus/content/groups/public/'
    ]) {
      def xmlTemplate = readFile templateFile
      String xmlFile = transformXml(xmlTemplate, env.nexusUsername, env.nexusPassword)

      String tempFile = 'temp.xml'
      writeFile file: tempFile, text: xmlFile

      sh "mvn -B clean build -s ${tempFile}"
    }
  }

The step withCredentials will not only provide a secure way of injecting secrets (e.g., Nexus credentials) into Jenkins pipeline, but also scrub away such sensitive information if we happen to print them out in log files. transformXml is my Groovy function that generates the settings.xml from the redacted Maven settings.xml template (no credentials) and the provided Nexus credentials.

Maven 3.0

Since Maven 3.0, the above problem is made much easier since environment variables can be referred inside settings.xml file by using special expression ${env.VAR_NAME}, based on this doc. Nexus authentication for Maven 3.0 in Jenkins pipeline can be done as follows:

settings.xml in Maven 3.0
1
2
3
4
5
6
7
8
9

<settings>
  <servers>
    <server>
      <id>nexus</id>
      <username>${env.MVN_SETTINGS_nexusUsername}</username>
      <password>${env.MVN_SETTINGS_nexusPassword}</password>
    </server>
  </servers>
</settings>
Passing Nexus credentials for Maven 3.0 in Jenkinsfile
1
2
3
4
5
6
7
8
9
10

  withCredentials([
    [$class: 'StringBinding', credentialsId: 'nexusUsername', variable: 'MVN_SETTINGS_nexusUsername'],
    [$class: 'StringBinding', credentialsId: 'nexusPassword', variable: 'MVN_SETTINGS_nexusPassword']
  ]) {
    withEnv([
      'nexusPublic=https://nexus.example.com/nexus/content/groups/public/'
    ]) {
      sh 'mvn -s settings.xml clean build'
    }
  }

However, note that it is still tricky even in Maven 3.0 since this is not always applicable, as noted in the same doc.

Note that properties defined in profiles within the settings.xml cannot be used for interpolation.

Gradle

In Gradle, Nexus authentication can be specified in both build.gradle and gradle.properties file, where build.gradle should be checked into VCS (e.g., git) while gradle.properties contains sensitive credentials information.

Example build.gradle
1
2
3
4
5
6
7
8
9

repositories {
    maven {
        credentials {
            username nexusUsername
            password nexusPassword
        }
        url { nexusPublic }
    }
}
Example gradle.properties
1
2
3

nexusUsername=myUsername
nexusPassword=password123
nexusPublic=https://nexus.example.com/nexus/content/groups/public/

The default location of the gradle.properties file is ~/.gradle. This is due to the environment variable GRADLE_USER_HOME usually set to ~/.gradle. For custom location of gradle.properties (i.e., other than ~/.gradle), ensure that the variable GRADLE_USER_HOME is set accordingly.

However, similar to Maven, for Jenkins pipeline automation, it is not safe to store credentials in plain text file gradle.properties, no matter how “hidden” its location is. For that purpose, you should use the following Groovy code:

Nexus authentication for Gradle in Jenkinsfile.
1
2
3
4
5
6
7
8
9
10
11
12

  withCredentials([
    [$class: 'StringBinding', credentialsId: 'nexusUsername', variable: 'ORG_GRADLE_PROJECT_nexusUsername'],
    [$class: 'StringBinding', credentialsId: 'nexusPassword', variable: 'ORG_GRADLE_PROJECT_nexusPassword']
  ]) {
    withEnv([
      'ORG_GRADLE_PROJECT_nexusPublic=https://nexus.example.com/nexus/content/groups/public/',
      'ORG_GRADLE_PROJECT_nexusReleases=https://nexus.example.com/nexus/content/repositories/releases',
      'ORG_GRADLE_PROJECT_nexusSnapshots=https://nexus.example.com/nexus/content/repositories/snapshots'
    ]) {
      sh './gradlew jenkinsBuild'
    }
  }

Note that, in Gradle, the solution is much simpler because Gradle respects properies set through environment variales. Based on its doc, if the environment variable name looks like ORG_GRADLE_PROJECT_prop=somevalue, then Gradle will set a prop property on your project object, with the value of somevalue. Therefore, in withCredentials step, we specifically bind the secrets nexusUsername and nexusPassword to the environment variables ORG_GRADLE_PROJECT_nexusUsername and ORG_GRADLE_PROJECT_nexusPassword and not some arbitrary variable names. These environment variables should match the ones used in builde.gradle and, in the following Closure, we simply call the standard Gradle wrapper command ./gradlew <target>. Compared with Maven solution in the last section, there is no intermediate step to generate settings.xml based on the provided secrets.

More Tips

If Maven/Gradle build is used in multiple repositories across organization, it is recommended to move the above Groovy code into shared Jenkins library, as shown in last post. For example, the Gradle builds can be simplified by defining useNexus step (see here) and adding it into the shared library workflow-lib.

vars/useNexus.groovy
1
2
3
4
5
6
7
8
9
10
11
12
13
14

def call(Closure body) {
  withCredentials([
    [$class: 'StringBinding', credentialsId: 'nexusUsername', variable: 'ORG_GRADLE_PROJECT_nexusUsername'],
    [$class: 'StringBinding', credentialsId: 'nexusPassword', variable: 'ORG_GRADLE_PROJECT_nexusPassword']
  ]) {
    withEnv([
      'ORG_GRADLE_PROJECT_nexusPublic=https://nexus.example.com/nexus/content/groups/public/',
      'ORG_GRADLE_PROJECT_nexusReleases=https://nexus.example.com/nexus/content/repositories/releases',
      'ORG_GRADLE_PROJECT_nexusSnapshots=https://nexus.example.com/nexus/content/repositories/snapshots'
    ]) {
      body()
    }
  }
}

After that, all the Gradle builds with Nexus authentication in Jenkinsfile will now be reduced to simply this:

Simplified Nexus authentication for Gradle in Jenkinsfile.
1
2
3

useNexus {
  sh './gradlew jenkinsBuild'
}

As shown above, it will reduce lots of redundant codes for Gradle builds, repeated again and again in Jenkinsfiles across multiple repositories in an organizaiton.

References

]]>