It took me quite sometime to learn the ideosyncracy of withPythonEnv. If you are trying to do something similar to mine, I hope this post helps you. I went through normal channels like StackOverflow.com, Meidum and Google search, I didn’t find much help so I had to bang my head on keyboard to try&error for many times.

Background

First, the prodject structure. This is a Django project. It seems to me that the most of Django project directory found on the internet puts the Django’s directory as root of project. This is pretty unrealistic. On the toplevel, you want to have non-web related things like README, the housekeeping shell scripts, and testing fixures outside of it. The most notable file for this is my “passenger” file which the website DreamHost provides to run WSGI. This is essential to setting up the Django before diving into Django app.

MyProject
  - passenger_wsgi.py
  - DjangoApp
    - DjangoApp
      - settings.py
  - Housekeeping
  - Makefile
  - venv
  - etc.

This Makefile is used to set up Python virtual environment (venv). In order to run the Django app, it needs django and other packages. venv gets created and populated by necessary packages by this Makefile. I wanted vent to be right next to DjangoApp so the passenger_wsgi.py can set up the virtual env and then run the Django requests.
I therefore wanted to do the same for Jenkins. I looked at the withPythonEnv, I thought I can run “make venv” and off I go. Use withPythonEnv(‘venv/bin/python3’) allows me to run Django with Jenkins workspace once I bootstrap the “venv”. I was very wrong.
I create the virutal env, then try to use, but it had different idea. It used the Python I designated to set up its own venv in the workspace.
Also, this means, I cannot use single Makefile verb to do the deployment and Jenkins workspace. I attempted a few things like making Makefile to take care of build/test, dot-include venv/bin/activate for every “sh” comamnd in the jenkinsfile stage. None of this worked.
Using “sh” was most surprising for a noob like me. It created a temp sh file and run it in a sub/sub directory so any of working directory relative includes and executions work.
IOW, I tried to do it without withPythonEnv, and I could not find a good way other than write everything in Makefile and let make to do the work including test. I did not like this idea.
So, I went with withPythonEnv, and ate up the limitation that comes with it. It creates ProjectRoot/.pyenv-<Python> where is the name of Python in Global Tools. I also tried a few permutation of entries in Global Tools, and after some frustrating attempts, I ended up having just a very plain entry.

Python in Jenkins Global Tools Configuration

Albeit this is named for the project, the entry is the most basic.

  - Python
    Name: Python3
    Home or executable: /usr/bin/python3

That’s it. Ignore the yellow warning for the home or executable part. I tried other things like auto-install, and didn’t make sense of it. As withPythonEnv creating its own venv, using venv in Global Tools Configuration makes very little sense. You do need to install “venv” package to the system before doing this. If you don’t have the perm to do so, but if you can become “jenkins” user, you could install “venv using pip so it’s installed under the home directory of user “jenkins”. In other word, using virutal env for this entry makes sense only if you have no root perm or jenkins auth so you must create your own venv to create venv. Since python3 should exist, the simplest solution is to use python3, and install venv package for it.

Jenkinsfile for the project

Here is my Jenkinsfile now.

pipeline {
    agent any

    environment {
        PYTHONPATH = "${env.WORKSPACE}/cworg"
        DJANGO_SETTINGS_MODULE='DjangoApp.settings'
        BUILD_NUMBER = "${env.BUILD_NUMBER}"

        GIT_URL="ntai@git.my-server.com:~/git/MyProject.git"
    }

    options {
        buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '', daysToKeepStr: '10', numToKeepStr: '20'))
        timestamps()
        retry(1)
        timeout time:10, unit:'MINUTES'
    }

    parameters {
        string(defaultValue: "master", description: 'Branch Specifier', name: 'SPECIFIER')
    }

    stages {
        stage("Initialize") {
            steps {
            script {
                    echo "${BUILD_NUMBER} - ${env.BUILD_ID} on ${env.JENKINS_URL}"
                    echo "Branch Specifier :: ${params.SPECIFIER}"
            }
            }
        }

        stage('Checkout') {
            steps {
            git branch: "${params.SPECIFIER}", url: "${GIT_URL}"
            }
        }

        stage('Make Virtual Env') {
            steps {
                withPythonEnv('Python3') {
                    sh 'pip install -r requirements.txt'
                }
            }
        }

        stage('Bootstrap') {
            steps {
            dir ("cworg") {
                withPythonEnv('Python3') {
                sh "make bootstrap"
                }
                }
            }
        }

        stage('Build') {
            steps {
            dir("cworg") {
                withPythonEnv('Python3') {
                        sh "make static"
                }
                }
            }
        }

        stage('Test') {
            steps {
                dir('./') {
                withPythonEnv('Python3') {
                        sh "python3 -m pytest"
                }
                }
            }
        }

        stage('Deploy') {
        steps {
        sh "ssh webapp@my-server.com'~/deploy-my-django-app.sh'"
        }
        }
    }
}

With using ‘Python3’ tool thing, it creates jenkins/workspace/MyProject/.pyenv-Python3 I can now run the test with the virutal env and against the test database for the Django project.