Artifact Deployments

travis-repositories-diagram
Plenty of services in the title of this blog, so let’s start with looking at the JVM repository landscape.

Probably everyone reading this post knows what Maven Central is. But for those who do not, it is probably the most commonly used artifact repository out there. All your build tools will most definitely use it out of the box (Maven, Gradle, SBT…).

Now for opensource projects, the way to put your artifacts into Maven Central is upload to OSS Sonatype repository that gets synchronised with Maven Central (in ~10 minutes after deployment).

So in theory the above should be enough, but technology hates monopolies, so if you want to be cool these days, you should use Bintray – the not so new service, which killer feature is that it doesn’t look like designed in 1997 (Sonatype, really? why does nexus have to look that bad?). Bintray itself will just give you a set of repositories for free for you OSS project, but the cool thing is that it can link it’s repos (which basically means artifact synchronisation).

The most common place to link your Bintray repos to is jCenter. It aims to be THE replacement for Maven Central. And it kinda is getting there. It is used very often for gradle dependencies, it is a standard place for sbt plugins. Your build tools will have jCenter urls hardcoded (in sbt there is Resolvers.jCenter; in gradle repositories { jcenter() }).

Now Bintray allows only non-SNAPSHOT releases, so if you’d like to deploy your opensource SNAPSHOT artifacts somewhere, you can apply for a JFrog OSS account through Bintray. It neatly integrates with Bintray, so that with a click of a button you can promote your snapshot release to a Bintray major release.

The last puzzle is Travis CI which is a Continuous Integration tool (hence the CI). Not only it is very cool, configured in a versioned file that lives with your source code, but it’s free for opensource.

What we would like to achieve now, is to have our project built by Travis CI automatically and if version is SNAPSHOT then it should be deployed to JFrog OSS and for non-SNAPSHOT it should deploy to Bintray, JCenter and Maven Central (why not 😉 our library deserves to be downloaded!).

The whole reason for this post is that a while ago I have started with Adam Warski a new scala frontend library called Supler. And it does all described above.

Prerequisites

There are few things you need to get in place before we can start setting up sbt.

Sonatype OSS account

Create yourself a Sonatype account following the guide here http://central.sonatype.org/pages/ossrh-guide.html

You can skip this step if Bintray is enough for you, and you do not need to promote your binaries to maven central.

Bintray account

First go to http://bintray.com and register if you haven’t done it yet.

Then we will need to setup a new package (which basically is you application).

Setup your bintray project

Login and choose maven repository (or create a new one if you wish to)

maven-repo

Create new package

create-repo-button

and provide all the needed details

create-repo-screen

Go to your package screen and link it to JCenter, asking for JFrog OSS account

link-button

remember to check “Host my snapshot build artifacts on the OSS Artifactory … ”

link-screen

Once your request is accepted you will see it linked to jcenter and (AFAIR) you will get a message, that your JFrog OSS account is ready

linked-supler

Then go to your profile settings and setup your Sonatype username

sonatype_user

And post your GPG keys that you will use for signing jars (or not, if you do not want to share them, then your process wont be automatic 🙂 )

edit-gpg

Deployments to Bintray and JFrog OSS

Now the build.sbt setup.

Add bintray-sbt plugin to your project/plugins.sbt

resolvers += Resolver.jcenterRepo

addSbtPlugin("me.lessis" % "bintray-sbt" % "0.3.0")

Add publish settings to build.sbt

val Version = "0.5.0-SNAPSHOT"

val buildSettings = Defaults.coreDefaultSettings ++ Seq(
    version := Version,
    scalaVersion := "2.11.5",
    scalacOptions ++= Seq("-unchecked", "-deprecation", "-feature", "-language:existentials", "-language:higherKinds"),
    parallelExecution := false
  )

val doNotPublishSettings = Seq(publish := {})

val publishSettings =
    (if (Version.endsWith("-SNAPSHOT"))
      Seq(
        publishTo := Some("Artifactory Realm" at "http://oss.jfrog.org/artifactory/oss-snapshot-local"),
        bintrayReleaseOnPublish := false,
        // Only setting the credentials file if it exists (#52)
        credentials := List(Path.userHome / ".bintray" / ".artifactory").filter(_.exists).map(Credentials(_))
      )
    else
      Seq(
      organization := "YOUR.GROUP",
      pomExtra := <scm>
        <url>YOUR.SCM</url>
        <connection>YOUR.SCM</connection>
      </scm>
        <developers>
          <developer>
            <id>john</id>
            <name>John Doe</name>
            <url>http://www.does.org/~john</url>
          </developer>
        </developers>,
      publishArtifact in Test := false,
      homepage := Some(url("YOUR.URL")),
      publishMavenStyle := false,
      resolvers += Resolver.url("supler ivy resolver", url("http://dl.bintray.com/YOUR_BINTRAY_USER/maven"))(Resolver.ivyStylePatterns),
      licenses := ("Apache-2.0", url("http://www.apache.org/licenses/LICENSE-2.0.txt")) :: Nil // this is required! otherwise Bintray will reject the code
    )

Now when you setup your subprojects add publishSettings for the projects that should be published and doNotPublishSettings for those that should be omitted.

You will see that version is set as a val – it is just easier to IF on it and check if the version is a SNAPSHOT version or not. You will see that when the version *is* SNAPSHOT then the bintray release process will be switched off and JFrog OSS Artifactory will be used instead.

Now you might’ve noticed something weird in the above code – publishMavenStyle is set to false and there’s some weird resolver added – this is apparently some recent Bintray bug described here: http://stackoverflow.com/questions/31704818/releasing-and-publishing-from-sbt-bintray – once this is fixed, i’ll update this post.

The project might look like this. Notice the publishSettings have been added.

lazy val supler: Project = Project(
    "supler",
    file("supler"),
    settings = buildSettings ++ publishSettings ++ Seq(
      libraryDependencies <+= (scalaVersion)("org.scala-lang" % "scala-compiler" % _ % "provided"),
      libraryDependencies ++= Seq(json4sNative, scalaTest))
  )

Provide credentials

Create ~/.bintray/.credentials and ~/.bintray/.artifactory files and fill them in with your user and pass

~/.bintray/.credentials

realm = Bintray API Realm
host = api.bintray.com
user = YOUR.USER
password = YOUR.PASSWORD which you can get in your profile under API Key

~/.bintray/.artifactory

realm = Artifactory Realm
host = oss.jfrog.org
user = SAME.BINTRAY.USER
password = SAME.BINTRAY.PASSWORD

Try it out

Go to your project and type

sbt publish

then go and change the version from/to -SNAPSHOT and see that it uploads to the correct place.

Travis

Basic Travis setup

The next step will be to set it all up on Travis CI.

We need to start with a .travis.yml file that defines the project

language: scala
sudo: false
scala:
- 2.11.5

The file should be self-describing but the full reference of the .travis.yml can be found here: http://docs.travis-ci.com/user/customizing-the-build/

Add this file to the root of your build, and then login to travis-ci and enable your project to be picked up. Once this is green, we’re good to go.

Encode passwords

Now we are going to need to encode few passwords, to store them in the .travis.yml file. If you do not feel comfortable with storing your password (even encoded) on a public github repository, then the only way to go will be having your private build system, but you should not be very worried – Travis creates a set of private/public keys for each project so you should be the only one having access to your secure variables. Moreover your secure variables will not be available to Pull Requests from forks, so the guys from Travis seem to though it through. You can read more on storing those secured variables here.

Start with installing travis CLI

gem install travis

And then you will need to encrypt BINTRAY_USER, BINTRAY_PASSWORD (the API Key actually), SONATYPE_USER and SONATYPE_PASSWORD (for maven central synchronisations). You do it by running the below command in your root directory (where .travis.yml) resides.

$ travis encrypt BINTRAY_USER=YOUR.user --add
$ travis encrypt BINTRAY_PASSWORD=YOUR.API.KEY --add
$ travis encrypt SONATYPE_USER=YOUR.SONATYPE.user --add
$ travis encrypt SONATYPE_PASSWORD=YOUR.SONATYPE.pass --add

Once you have all vars encyrpted your .travis.yml should look like this:

language: scala
sudo: false
scala:
- 2.11.5
env:
  global:
  - secure: rI0yiqoisNVKhH/ofJ00Ymj47Uw4FYvxhiNmCv5LObfu6dQQohaMi9BAVzpr+aVvBeXfWT5As9giTuDZfcxkHoXXMM0VIEgp5XpLxRFDOJBmvgagJqN/tM22yU3aDexQm/ORBpETLnYtvbazbo6q7fYT+pfc8HEtu8BfbpLe4ro=
  - secure: f2iy9eo7ZQThWGNW1kCllTEFaGi9EuiJKYnRe6SsbXZjmxeo8Q+KvPvbqOjkvdH43krFySnrF1VO/fdhDMcP2XmnCkyyX37K/c/iOZSWoyCydYeydM0YgNoUpoxiFC9WCgffSWI4q1Tn0RsC+yzXrysXf6/ii4S8SBuRv4yBTHA=
  - secure: f3IJrafmfHgCY5HNep4/jO6K0KBpc4wkKm4yG3sK/lzcRS0oD9t1/rY1YbO7msUcIp4YcLOVJvBIKIwfNSI5yA+5RTogeW1LllE5WawvxsRPfeH1MJneb/eTpRzBjAVyK/9u3l9KzuPt+GDJuILK2Sc7XaihKj5mhXmQ5mI2aoI=
  - secure: FTcC++wfkERq/HWopJQyG6BctvhUUw2EySyshksN3B5aVekztT+fx55OFM0BmZ75EyGhq4r0SFJvirCcmq9+uUwQse++OqZbNEGw/71FB55FD1Wzo5Xg+XaO6FS2YU+J58dPWhBTHCurP+ETqxFz/oPIOAPbHSIvGsvESlLshrA=

And once the travis build starts you will see similar message at the beginning:

Setting environment variables from .travis.yml
$ export BINTRAY_USER=[secure]
$ export BINTRAY_PASSWORD=[secure]
$ export SONATYPE_PASSWORD=[secure]
$ export SONATYPE_USER=[secure]

Publishing and Maven Central Synchronisation

With the password encoded we are ready to add the actual synchronisation and publishing.

Firstly create a new bash script that will create the ~/.bintray/.credentials and ~/.bintray/.artifactory during our Travis build

.travis.credentials.sh

#!/bin/bash
mkdir ~/.bintray/
BINTRAY_FILE=$HOME/.bintray/.credentials
ARTIFACTORY_FILE=$HOME/.bintray/.artifactory
cat <<EOF >$BINTRAY_FILE
realm = Bintray API Realm
host = api.bintray.com
user = $BINTRAY_USER
password = $BINTRAY_PASSWORD
EOF

cat <<EOF >$ARTIFACTORY_FILE
realm = Artifactory Realm
host = oss.jfrog.org
user = $BINTRAY_USER
password = $BINTRAY_PASSWORD
EOF

Then add a simple task to build.sbt that will create a script file with the project version set (you should .gitignore .run.central.synchro.sh)

lazy val makeVersionSh = taskKey[Seq[File]]("Creates .run.central.synchro.sh file.")

lazy val root: Project = Project(
  "root",
  file("."),
  settings = buildSettings ++ doNotPublishSettings ++ Seq(
    publishArtifact := false,
    makeVersionSh := {
      val pf = new java.io.File(".run.central.synchro.sh")
      val content = s"""|#!/bin/bash
                       |PROJECT_VERSION=${version.value} /bin/bash .central.synchro.sh
                      """.stripMargin
      IO.write(pf, content)
      Seq(pf)
    }
  )
) aggregate(supler, suplerjs, examples)

This will create a file with a shebang, PROJECT_VERSION set and call to run .central.synchro.sh, which should look like this (put it in your project’s root directory):

.central.synchro.sh

#!/bin/bash
if [[ ${PROJECT_VERSION} != *-SNAPSHOT ]]
then
    curl -vvf -u$BINTRAY_USER:$BINTRAY_PASSWORD -H "Content-Type: application/json" \
        -X POST https://bintray.com/api/v1/maven_central_sync/softwaremill/softwaremill/supler/versions/${PROJECT_VERSION} \
        --data "{ \"username\": \"${SONATYPE_USER}\", \"password\": \"${SONATYPE_PASSWORD}\", \"close\": \"1\" }"
    echo "Release synchronized to central"
else
    echo "Ignoring snapshot version"
fi

The above calls a special REST API on Bintray that promotes the binaries to the Maven Central via Sonatype OSS.

Finally update .travis.yml to run the above scripts

language: scala
sudo: false
scala:
- 2.11.5
script:
- sbt ++$TRAVIS_SCALA_VERSION test
after_success:
- '[ "$TRAVIS_BRANCH" = "master" -a "$TRAVIS_PULL_REQUEST" = "false" ] && /bin/bash "$TRAVIS_BUILD_DIR/.travis.credentials.sh" && sbt publish && sbt makeVersionSh && /bin/bash .run.central.synchro.sh'
env:
  global:
  - secure: rI0yiqoisNVKhH/ofJ00Ymj47Uw4FYvxhiNmCv5LObfu6dQQohaMi9BAVzpr+aVvBeXfWT5As9giTuDZfcxkHoXXMM0VIEgp5XpLxRFDOJBmvgagJqN/tM22yU3aDexQm/ORBpETLnYtvbazbo6q7fYT+pfc8HEtu8BfbpLe4ro=
  - secure: f2iy9eo7ZQThWGNW1kCllTEFaGi9EuiJKYnRe6SsbXZjmxeo8Q+KvPvbqOjkvdH43krFySnrF1VO/fdhDMcP2XmnCkyyX37K/c/iOZSWoyCydYeydM0YgNoUpoxiFC9WCgffSWI4q1Tn0RsC+yzXrysXf6/ii4S8SBuRv4yBTHA=
  - secure: f3IJrafmfHgCY5HNep4/jO6K0KBpc4wkKm4yG3sK/lzcRS0oD9t1/rY1YbO7msUcIp4YcLOVJvBIKIwfNSI5yA+5RTogeW1LllE5WawvxsRPfeH1MJneb/eTpRzBjAVyK/9u3l9KzuPt+GDJuILK2Sc7XaihKj5mhXmQ5mI2aoI=
  - secure: FTcC++wfkERq/HWopJQyG6BctvhUUw2EySyshksN3B5aVekztT+fx55OFM0BmZ75EyGhq4r0SFJvirCcmq9+uUwQse++OqZbNEGw/71FB55FD1Wzo5Xg+XaO6FS2YU+J58dPWhBTHCurP+ETqxFz/oPIOAPbHSIvGsvESlLshrA=

The key here is the after_success section. It also has a check that we are not build a Pull Request, and we are on the master branch (I do not want to deploy other branches to maven repos).

Summary

By now you should have a working solution that will build your project on Travis CI after each commit, publishing to JFrog OSS repos or Bintray/jCenter/Maven Central when changes are committed to the master branch.

You can check out the full working example at https://github.com/softwaremill/supler

Automatic deployments to JFrog OSS and Bintray/jCenter/Maven Central via Travis CI from SBT
Tagged on:                         

3 thoughts on “Automatic deployments to JFrog OSS and Bintray/jCenter/Maven Central via Travis CI from SBT

  • September 30, 2015 at 10:38 pm
    Permalink

    There is some kind of syntax error in “val publishSettings = …”, at least one bracket is missing.

    I suppose there should be something like this:


    credentials := List(Path.userHome / “.bintray” / “.artifactory”).filter(_.exists).map(Credentials(_))
    )
    else Seq()) ++
    Seq(
    organization := “YOUR.GROUP”,

    Reply
  • November 26, 2015 at 11:43 pm
    Permalink

    By Jerome Ruillier October 16, 2012 – 7:11 amGreat article, ecaltxy what I was looking for !I also found that Red Hat, through OpenShift, offers a great tool combination :Git + Jenkins + Nexus (since September 27, 2012)Surprisingly, it’s very difficult to find any company offering those three services together. Regards,Jerome

    Reply
  • Pingback: Easy publishing to Maven Central with Gradle | :: (Bloggable a) => a -> IO ()

Leave a Reply

Your email address will not be published. Required fields are marked *