Configuration file¶

The configuration file is read by the scikit-ci executable to find out which commands to execute for a given step.

The configuration file should named scikit-ci.yml and is usually added to the root of a project.

It is a YAML file that can be validated against scikit-ci-schema.yml.

Concept of Step¶

A step consist of a list of commands and optional key/value pairs describing the environment.

More specifically, a step can be described using the following structure:

before_install:
  environment:
    FOO: bar
  commands:
    - echo "Hello world"

where before_install can be replaced by any of these:

before_install
install
before_build
build
test
after_test

Mapping with Appveyor, Azure Pipelines, CircleCI and TravisCI steps¶

scikit-ci do not impose any particular mapping.

Documentation specific to each services is available here:

Appveyor build pipeline

Azure pipelines

CircleCI configuration 2.0

CircleCI configuration 1.0 (deprecated)

TravisCI build lifecycle

Reported below are some recommended associations that are know to work.

appveyor.yml:

install:
  - python -m ci install

build_script:
  - python -m ci build

test_script:
  - python -m ci test

after_test:
  - python -m ci after_test

Note

Since on windows the ci executable is installed in the Scripts directory (e.g C:\Python27\Scripts\ci.exe) which is not in the PATH by default, the python -m ci syntax is used.

azure-pipelines.yml:

      - script: python -m ci install
        displayName: Install

      - script: python -m ci build
        displayName: Build

      - script: python -m ci test
        displayName: Test

      - powershell: |
          if ($env:CODECOV_TOKEN -ne $null -And $env:BUILD_REASON -ne "PullRequest") {
            python -m ci after_test
          }
        displayName: After Test
        env:
          CODECOV_TOKEN: $(CODECOV_TOKEN)

.circleci/config.yml (CircleCI 2.0):

  steps:
    - checkout
    - run:
        <<: *initialize-venv
    - run:
        name: Install scikit-ci
        command: |
          . ../venv/bin/activate
          BOOTSTRAP_BRANCH=$CIRCLE_BRANCH
          BOOTSTRAP_REPO_SLUG=$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME
          if [[ $CIRCLE_PR_USERNAME != "" ]]; then
            BOOTSTRAP_BRANCH=$(curl -s https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/pulls/${CIRCLE_PR_NUMBER} | jq -r '.head.ref')
            BOOTSTRAP_REPO_SLUG=$CIRCLE_PR_USERNAME/$CIRCLE_PR_REPONAME
          fi
          echo "BOOTSTRAP_BRANCH:$BOOTSTRAP_BRANCH"
          echo "BOOTSTRAP_REPO_SLUG:$BOOTSTRAP_REPO_SLUG"
          git clone git://github.com/$BOOTSTRAP_REPO_SLUG -b $BOOTSTRAP_BRANCH ../bootstrap-scikit-ci
          pip install -U ../bootstrap-scikit-ci
    - run:
        name: Install dependencies
        command: |
          . ../venv/bin/activate
          ci install
    - run:
        name: Flake8
        command: |
          . ../venv/bin/activate
          ci before_build
    - run:
        name: Build
        command: |
          . ../venv/bin/activate
          ci build
    - run:
        name: Test
        command: |
          . ../venv/bin/activate
          ci test
    - run:
        name: Coverage
        command: |
          . ../venv/bin/activate
          ci after_test

circle.yml (CircleCI 1.0):

dependencies:
  override:
    - |
      BOOTSTRAP_BRANCH=$CIRCLE_BRANCH
      BOOTSTRAP_REPO_SLUG=$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME
      if [[ $CIRCLE_PR_USERNAME != "" ]]; then
        BOOTSTRAP_BRANCH=$(curl -s https://api.github.com/repos/${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}/pulls/${CIRCLE_PR_NUMBER} | jq -r '.head.ref')
        BOOTSTRAP_REPO_SLUG=$CIRCLE_PR_USERNAME/$CIRCLE_PR_REPONAME
      fi
      echo "BOOTSTRAP_BRANCH:$BOOTSTRAP_BRANCH"
      echo "BOOTSTRAP_REPO_SLUG:$BOOTSTRAP_REPO_SLUG"
      git clone git://github.com/$BOOTSTRAP_REPO_SLUG -b $BOOTSTRAP_BRANCH ../bootstrap-scikit-ci
      pip install -U ../bootstrap-scikit-ci

    - ci install

test:
  override:
    - ci test

deployment:
  master:
    branch: master
    commands:
      - ci after_test

.travis.yml

install:
  - ci install

script:
  - ci test

after_success:
  - ci after_test

Order of steps¶

scikit-ci execute steps considering the following order:

before_install
install
before_build
build
test
after_test

This means that the mapping specified in the continuous integration file has to be done accordingly.

Automatic execution of dependent steps¶

Considering the step ordering, executing any step(n) ensures that step(n-1) has been executed before.

Keeping track of executed steps¶

scikit-ci keeps track of executed steps setting environment variables of the form SCIKIT_CI_<STEP_NAME> where <STEP_NAME> is any of the step name in upper-case.

Note

Specifying the command line option --force allows to force the execution of the steps ignoring the values of the SCIKIT_CI_<STEP_NAME> environment variables.

Environment variable persistence¶

Environment variable defined in any given step are always guaranteed to be set in steps executed afterward.

This is made possible by serializing the environment on the filesystem.

Note

After executing steps, a file named env.json is created in the current directory along side scikit-ci.yml. This is where the environment is cached for re-use in subsequent steps.

Specifying the command line option --clear-cached-env allows to execute steps after removing the env.json file.

Step specialization¶

For any given step, it is possible to specify commands and environment variables specific to each continuous integration service.

Recognized services are:

appveyor

azure

circle

travis

Commands¶

commands common to all services are executed first, then commands specific to each services are executed.

For example, considering this configuration used on CircleCI and TravisCI:

before_install:
  commands:
    - echo "Hello Everywhere"

  circle:
    commands:
      - echo "Hello on CircleCI"

  travis:
    linux:
      commands:
        - echo "Hello on TravisCI"

The output on the different service will be the following:

CircleCI:

Hello Everywhere
Hello on CircleCI

TravisCI:

Hello Everywhere
Hello on TravisCI

Note

Sections Command Specification and Python Command Specification describe the different types of command.

Environment¶

Similarly, environment can be overridden for each service.

For example, considering this configuration used on CircleCI and TravisCI:

before_install:

  circle:
    environment:
      CATEGORY_2: 42

  travis:
    linux:
      environment:
        CATEGORY_1: 99

  environment:
    CATEGORY_1: 1
    CATEGORY_2: 2

  commands:
    - echo "CATEGORY_1 is ${CATEGORY_1}"
    - echo "CATEGORY_2 is ${CATEGORY_2}"

The output on the different service will be the following:

on CircleCI:

CATEGORY_1 is 1
CATEGORY_2 is 42

on TravisCI:

CATEGORY_1 is 99
CATEGORY_2 is 2

Reserved Environment Variables¶

CI_NAME: This variable is automatically set by scikit-ci and will contain the name of the continuous integration service currently executing the step.

Environment variable usage¶

To facilitate the use of environment variable across interpreters, scikit-ci uses a specific syntax.

Environment variable specified using $<NAME_OF_VARIABLE> in both commands and environment variable will be expanded.

For example, considering this configuration used on Appveyor, CircleCI and TravisCI:

before_install:

  appveyor:
    environment:
      TEXT: Windows$<TEXT>

  travis:
    linux:
      environment:
        TEXT: LinuxWorld

  environment:
    TEXT: World

  commands:
    - echo $<TEXT>

The output on the different service will be the following:

on Appveyor:
WindowsWorld
on CircleCI:
World
on TravisCI:
LinuxWorld

Note

On system having a POSIX interpreter, the environment variable will NOT be expanded if included in string start with a single quote.

class ci.driver.Driver[source]¶

static expand_command(command, environments, posix_shell=True)[source]¶

Return an updated command string where all occurrences of $<EnvironmentVarName> (with a corresponding env variable set) have been replaced.

If posix_shell is True, only occurrences of $<EnvironmentVarName> in string starting with double quotes will be replaced.

See https://www.gnu.org/software/bash/manual/html_node/Double-Quotes.html and https://www.gnu.org/software/bash/manual/html_node/Single-Quotes.html

Command Specification¶

Specifying command composed of a program name and arguments is supported on all platforms.

For example:

test:
  commands:
    - echo "Hello"
    - python -c "print('world')"
    - git clone git://github.com/scikit-build/scikit-ci

On unix based platforms (e.g CircleCI and TravisCI), commands are interpreted using bash.

On windows based platform (e.g Appveyor), commands are interpreted using the windows command terminal cmd.exe.

Since both interpreters expand quotes differently, we recommend to avoid single quoting argument. The following table list working recipes:

	CircleCi, TravisCI	Appveyor
scikit-ci command	bash output	cmd output
`echo Hello1`	Hello1	Hello1
`echo "Hello2"`	Hello2	“Hello2”
`echo 'Hello3'`	Hello3	‘Hello3’
`python -c "print('Hello4')"`	Hello4	Hello4
`python -c 'print("Hello5")'`	Hello5	`no output`
`python -c "print('Hello6\'World')"`	Hello6’World	Hello6’World

And here are the values associated with sys.argv for different scikit-ci commands:

python program.py --things "foo" "bar" --more-things "doo" 'dar'

Output on CircleCi, TravisCI:

arg_1 [--things]
arg_2 [foo]
arg_3 [bar]
arg_4 [--more-things]
arg_5 [doo]
arg_6 [dar]

Output on Appveyor:

arg_1 [--things]
arg_2 [foo]
arg_3 [bar]
arg_4 [--more-things]
arg_5 [doo]
arg_6 ['dar']    # <-- Note the presence of single quotes

python program.py --things "foo" "bar" --more-things "doo" 'dar'

Output on CircleCi, TravisCI:

arg_1 [--the-foo=foo]
arg_2 [-the-bar=bar]

Output on Appveyor:

arg_1 [--the-foo=foo]
arg_2 [-the-bar='bar']    # <-- Note the presence of single quotes

Note

Here are the source of program.py:

import sys
for index, arg in enumerate(sys.argv):
    if index == 0:
        continue
    print("arg_%s [%s]" % (index, sys.argv[index]))

Python Command Specification¶

New in version 0.10.0.

The python commands are supported on all platforms.

For example:

test:
  commands:
    - python: print("single_line")
    - python: "for letter in ['a', 'b', 'c']: print(letter)"
    - python: |
              import os
              if 'FOO' in os.environ:
                  print("FOO is set")
              else:
                  print("FOO is *NOT* set")

Note

By using os.environ, they remove the need for specifying environment variable using the $<NAME_OF_VARIABLE> syntax described in Environment variable usage.