Difference between revisions of " Release Guide"

From Blazegraph
Jump to: navigation, search
(Release)
 
(14 intermediate revisions by the same user not shown)
Line 5: Line 5:
 
== Preconditions ==
 
== Preconditions ==
  
 +
* Setup for Maven Central
 
* notify developers that we are closing the branch down for a release and wait for the all clear.
 
* notify developers that we are closing the branch down for a release and wait for the all clear.
 
* merge all tickets into master
 
* merge all tickets into master
 
* push snapshot to SF master
 
* push snapshot to SF master
 
* create release candidate branch.  BLAZEGRAPH_X_Y_Z_RC
 
* create release candidate branch.  BLAZEGRAPH_X_Y_Z_RC
* setup CI for RC branch
+
* Validate CI in branch
 +
* Update maven versions
 +
./scripts/updateVersion.sh X_Y_Z
 +
* edit scripts/version.properties to set the new version
 +
*Commit pom updates
 +
git commit -a -m "Release Version Updates for X_Y_Z"
 +
 
 
* verify performance against standard benchmarks : [[SPARQL_Benchmarks]]
 
* verify performance against standard benchmarks : [[SPARQL_Benchmarks]]
 
** [[LUBM]]
 
** [[LUBM]]
 
** [[BSBM]]
 
** [[BSBM]]
 
** Govtrack (see bigdata-perf/CI/govtrack)
 
** Govtrack (see bigdata-perf/CI/govtrack)
* verify artifact deployments:
+
* Build Artifacts (For verification)
** Verify executable jar
+
./scripts/buildArtifacts.sh
** Verify WAR deployment (tomcat).
+
* verify artifact deployments: For each artifact, start it with artifact-specific method and verify the version and git hash from the start-up log.
** Verify "ant deploy-artifact" deployment (jetty).
+
** Verify executable jar (java -jar bigdata.jar and java -jar blazegraph.jar)
* verify [http://www.bigdata.com/hudson/lastSuccessful/report/index.html CI builds].
+
** Verify WAR deployment (tomcat). (bigdata.war and blazegraph.war)
 +
** Verify Debian Deployer (Requires Ubuntu)
 +
** Verify RPM Deployer (Requires CentOS, RHEL, AMI)
 +
** Verify Tarball  (unzip to local directory and run bin/blazegraph.sh start)
 +
* verify [https://ci.blazegraph.com/ CI builds].
 
* Review release notes in [https://jira.blazegraph.com/projects/BLZG?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page JIRA].  Close or move any uncompleted tickets.
 
* Review release notes in [https://jira.blazegraph.com/projects/BLZG?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page JIRA].  Close or move any uncompleted tickets.
 
** write (and commit) release notes in bigdata/src/releases (make sure the TAG URL is reflected in the release notes!).  Copy and paste text version from JIRA and make any updates.
 
** write (and commit) release notes in bigdata/src/releases (make sure the TAG URL is reflected in the release notes!).  Copy and paste text version from JIRA and make any updates.
 
* Update the tag in Github with the release notes and binary artifacts
 
* Update the tag in Github with the release notes and binary artifacts
 +
  
  
Line 31: Line 43:
  
 
git remote add sourceforge ssh://beebs@git.code.sf.net/p/bigdata/git
 
git remote add sourceforge ssh://beebs@git.code.sf.net/p/bigdata/git
 +
git remote add blazegraph git@github.com:blazegraph/database.git
  
 
$ git remote -v
 
$ git remote -v
Line 37: Line 50:
 
sourceforge ssh://beebs@git.code.sf.net/p/bigdata/git (fetch)
 
sourceforge ssh://beebs@git.code.sf.net/p/bigdata/git (fetch)
 
sourceforge ssh://beebs@git.code.sf.net/p/bigdata/git (push)
 
sourceforge ssh://beebs@git.code.sf.net/p/bigdata/git (push)
 +
blazegraph git@github.com:blazegraph/database.git (fetch)
 +
blazegraph git@github.com:blazegraph/database.git (push)
  
 +
#If you need to delete and repush tags see.
 +
git tag -d 12345 #delete local
 +
git push origin :refs/tags/12345 #delete remote
 
</pre>
 
</pre>
  
 
== Release ==
 
== Release ==
  
* Bump the version in build.properties, make sure 'snapshot=false' and uncomment the "javadoc=" line so the javadoc will generate. Commit to GIT.
+
* Use the release.sh script with the new version number in the form X.Y.Z.  This script builds, creates a new git branch and tag, updates the POM versions, deploys to maven central, pushes the javadocs, and pushes a new branch for the reverse merge. MAKE SURE THAT YOUR KEY SIGNING CREDENTIALS ARE ENABLED.
* Create a TAG for the release in GIT. ** The TAG is the immutable snapshot of the code as of the release.  Maintenance will be done on the BRANCH from which the release was issued.  The TAG should use the ''tags'' path and have the general pattern ''.../bigdata/'''tags'''/BIGDATA_RELEASE_X_Y_Z''.
+
<pre>
 +
./scripts/release.sh 2.1.0
 +
</pre>
 +
 
 +
* Post Release
 +
** Deploy artifacts with the release version
 +
** Verify the artifacts are on [http://search.maven.org/#search%7Cga%7C1%7Ccom.blazegraph Maven Central]
 +
** Verify the [https://blazegraph.github.io/database/apidocs/index.html Javadocs] are online with the correct version.
 +
** Copy the contents of bigdata/artifacts/ into the Github tag as release artifacts.
 +
 
 +
 
 +
* (LEGACY:  Now done in release.sh) Create a TAG for the release in GIT. ** The TAG is the immutable snapshot of the code as of the release.  Maintenance will be done on the BRANCH from which the release was issued.  The TAG should use the ''tags'' path and have the general pattern ''.../bigdata/'''tags'''/BIGDATA_RELEASE_X_Y_Z''.
 
<pre>
 
<pre>
 
git push sourceforge BIGDATA_RELEASE_X_Y_Z
 
git push sourceforge BIGDATA_RELEASE_X_Y_Z
Line 51: Line 80:
 
</pre>
 
</pre>
 
* Create a new directory on sourceforge for the release [https://sourceforge.net/projects/bigdata/files/bigdata/].  Upload the release notes into the directory on sourceforge and rename as "README" (they will be displayed at the bottom of the folder listing on sourceforge once you do this and refresh the page view).   
 
* Create a new directory on sourceforge for the release [https://sourceforge.net/projects/bigdata/files/bigdata/].  Upload the release notes into the directory on sourceforge and rename as "README" (they will be displayed at the bottom of the folder listing on sourceforge once you do this and refresh the page view).   
* (Re-)generate the release artifacts and upload into the directory on sourceforge.  Be sure to click on the "i" icon and specify the necessary metadata for each release file (sourceforge will not update the "release" until you do this). DO NOT RENAME THESE DEPLOYMENT FILES.  The names you give to SF are the actual names for the downloaded files....
+
* Upload the artifacts generated by release.sh in the artifacts directory.
 
** Download these artifacts and verify that the downloaded files are the same (md5/md5sum).  There have been a lot of issues recently with sourceforge file uploads failing silently so you can not rely on the file being posted correctly the first time around.
 
** Download these artifacts and verify that the downloaded files are the same (md5/md5sum).  There have been a lot of issues recently with sourceforge file uploads failing silently so you can not rely on the file being posted correctly the first time around.
 
** "ant war" - generates bigdata.war (Standalone WAR deployment)
 
** "ant war" - generates bigdata.war (Standalone WAR deployment)
 
** "ant executable jar" -- generates bigdata-X_Y_Z-bundled.jar; rename to bigdata-bundled.jar (executable jar deployment)
 
** "ant executable jar" -- generates bigdata-X_Y_Z-bundled.jar; rename to bigdata-bundled.jar (executable jar deployment)
 
** "ant deploy-artifact" - generates REL.bigdata.''version''.tgz (HA deployment)
 
** "ant deploy-artifact" - generates REL.bigdata.''version''.tgz (HA deployment)
** "ant javadoc" - generates javadoc.  Upload to web site using rsync (see 'bigdata-project-release.txt' in dropbox for details), and verify the [http://www.blazegraph.com/docs/api/ javadoc] online.
+
** (LEGACY)  "ant javadoc" - generates javadoc.  Upload to web site using rsync (see 'bigdata-project-release.txt' in dropbox for details), and verify the [http://www.blazegraph.com/docs/api/ javadoc] online.
*** Rsync:  rsync -qrzPl --perms --chmod=ugo+X,ugo+r -e ssh ant-build/docs www.blazegraph.com:/var/www/html/www.blazegraph.com/files/
+
*** (LEGACY)  Rsync:  rsync -qrzPl --perms --chmod=ugo+X,ugo+r -e ssh ant-build/docs www.blazegraph.com:/var/www/html/www.blazegraph.com/files/
** "ant ant-install-artifact" - generates a source code distribution. See 'bigdata-project-release.txt' in dropbox for '''important''' details on how and where to install this.
+
** (LEGACY)  "ant ant-install-artifact" - generates a source code distribution. See 'bigdata-project-release.txt' in dropbox for '''important''' details on how and where to install this.
** Publish bigdata JAR to the [http://www.systap.com/maven/ systap maven repository]:
+
** (LEGACY)  Publish bigdata JAR to the [http://www.systap.com/maven/ systap maven repository]:
 
*** NOTE:  If the fastutil version has changed you will need to upload this to the maven repository.  See comments in the POM.xml file.
 
*** NOTE:  If the fastutil version has changed you will need to upload this to the maven repository.  See comments in the POM.xml file.
*** Change the version in pom.xml (bump version number, remove "-SNAPSHOT", do NOT commit).
+
*** (LEGACY)  Change the version in pom.xml (bump version number, remove "-SNAPSHOT", do NOT commit).
*** "mvn clean deploy" - deploys the new bigdata jar to the [http://www.systap.com/maven/ systap maven repository] (alternatively try 'ant maven-deploy").  
+
*** (LEGACY)  "mvn clean deploy" - deploys the new bigdata jar to the [http://www.systap.com/maven/ systap maven repository] (alternatively try 'ant maven-deploy").  
*** Append "-SNAPSHOT" to the version pom.xml and commit to resume snapshot releases in CI.
+
*** (LEGACY)  Append "-SNAPSHOT" to the version pom.xml and commit to resume snapshot releases in CI.
** Review the [http://www.bigdata.com/download download page] and edit as necessary.
+
** Review the [http://www.blazegraph.com/download download page] and edit as necessary.
 
** TODO:  Document process to snapshot user guide / wiki.
 
** TODO:  Document process to snapshot user guide / wiki.
  
Line 76: Line 105:
 
*** SF master, github master should be equal.
 
*** SF master, github master should be equal.
 
*** SF release tag, github release tag should be equal.
 
*** SF release tag, github release tag should be equal.
* for major releases (1.1.0, 1.2.0, etc).
 
** create a new development branch (''.../bigdata/branches/BIGDATA_RELEASE_X_Y_Z'').
 
** notify developers to switch to the new development branch.
 
** update CI to run against the new development branch (create a new hudson job from the old development branch job).
 
 
* notify developers that the branch is open again.
 
* notify developers that the branch is open again.
 
* blog it [http://www.bigdata.com/blog] and elsewhere.
 
* blog it [http://www.bigdata.com/blog] and elsewhere.
 
* post the release notes to the sourceforge project news (logged in as an admin, choose "News" from the tabs and then "Post New")
 
* post the release notes to the sourceforge project news (logged in as an admin, choose "News" from the tabs and then "Post New")
 +
* Update the website download link
 +
* Make sure the new files are set as the default download on Sourceforge
 
* send out an email announcement (we maintain a mailing list).
 
* send out an email announcement (we maintain a mailing list).
 
* send out a tinkerpop mailing list announcement
 
* send out a tinkerpop mailing list announcement
Line 97: Line 124:
  
 
There is a pom.xml file in the top-level of the bigdata project that controls the publication of the bigdata releases and snapshots.  This pom must be updated any time a dependency version is updated or a new dependency is added.  The pom also includes instructions (in XML comment blocks) on how to publish non-bigdata artifacts to the bigdata maven repository. This is used to bootstrap non-bigdata maven artifacts that are not otherwise widely available.  In addition, any time we create a new release for one of the sub-projects (bigdata-ganglia, etc.), the new release artifact needs to be pushed to the maven repository.
 
There is a pom.xml file in the top-level of the bigdata project that controls the publication of the bigdata releases and snapshots.  This pom must be updated any time a dependency version is updated or a new dependency is added.  The pom also includes instructions (in XML comment blocks) on how to publish non-bigdata artifacts to the bigdata maven repository. This is used to bootstrap non-bigdata maven artifacts that are not otherwise widely available.  In addition, any time we create a new release for one of the sub-projects (bigdata-ganglia, etc.), the new release artifact needs to be pushed to the maven repository.
 +
 +
=== Maven Central Guide ===
 +
 +
[https://maven.apache.org/guides/mini/guide-central-repository-upload.html Maven Central]
 +
[http://central.sonatype.org/pages/apache-maven.html Maven Deployment]
 +
 +
=== PGP Key Setup ===
 +
 +
[https://www.phildev.net/pgp/gpg_moving_keys.html Here] is a guide to moving existing PGP keys.
 +
 +
=== Encrypted Passwords ===
 +
 +
Setup your maven installation to have encrypted passwords [https://maven.apache.org/guides/mini/guide-encryption.html#How_to_create_a_master_password Maven Password Encryption].

Latest revision as of 18:23, 9 June 2016

This is meant to be a living document for performing releases of the bigdata project. See [1] for discussions about this page.

Project releases are done as branches in GIT. The branch name has the general form BIGDATA_RELEASE_X_Y_Z which corresponds to version X.Y.Z. This way the project name makes it into the tag and hence into the directory when it is checked out from GIT. Use a branch for the release since there is an expectation that we will have to support the branch.

Preconditions

  • Setup for Maven Central
  • notify developers that we are closing the branch down for a release and wait for the all clear.
  • merge all tickets into master
  • push snapshot to SF master
  • create release candidate branch. BLAZEGRAPH_X_Y_Z_RC
  • Validate CI in branch
  • Update maven versions
./scripts/updateVersion.sh X_Y_Z
  • edit scripts/version.properties to set the new version
  • Commit pom updates
git commit -a -m "Release Version Updates for X_Y_Z"
  • verify performance against standard benchmarks : SPARQL_Benchmarks
    • LUBM
    • BSBM
    • Govtrack (see bigdata-perf/CI/govtrack)
  • Build Artifacts (For verification)
./scripts/buildArtifacts.sh
  • verify artifact deployments: For each artifact, start it with artifact-specific method and verify the version and git hash from the start-up log.
    • Verify executable jar (java -jar bigdata.jar and java -jar blazegraph.jar)
    • Verify WAR deployment (tomcat). (bigdata.war and blazegraph.war)
    • Verify Debian Deployer (Requires Ubuntu)
    • Verify RPM Deployer (Requires CentOS, RHEL, AMI)
    • Verify Tarball (unzip to local directory and run bin/blazegraph.sh start)
  • verify CI builds.
  • Review release notes in JIRA. Close or move any uncompleted tickets.
    • write (and commit) release notes in bigdata/src/releases (make sure the TAG URL is reflected in the release notes!). Copy and paste text version from JIRA and make any updates.
  • Update the tag in Github with the release notes and binary artifacts


  • Configured your git repository to be able to push to the remote sourceforge repository
$ git remote -v
origin	git@github.com:SYSTAP/bigdata.git (fetch)
origin	git@github.com:SYSTAP/bigdata.git (push)

git remote add sourceforge ssh://beebs@git.code.sf.net/p/bigdata/git
git remote add blazegraph git@github.com:blazegraph/database.git

$ git remote -v
origin	git@github.com:SYSTAP/bigdata.git (fetch)
origin	git@github.com:SYSTAP/bigdata.git (push)
sourceforge	ssh://beebs@git.code.sf.net/p/bigdata/git (fetch)
sourceforge	ssh://beebs@git.code.sf.net/p/bigdata/git (push)
blazegraph	git@github.com:blazegraph/database.git (fetch)
blazegraph	git@github.com:blazegraph/database.git (push)

#If you need to delete and repush tags see.
git tag -d 12345 #delete local
git push origin :refs/tags/12345 #delete remote

Release

  • Use the release.sh script with the new version number in the form X.Y.Z. This script builds, creates a new git branch and tag, updates the POM versions, deploys to maven central, pushes the javadocs, and pushes a new branch for the reverse merge. MAKE SURE THAT YOUR KEY SIGNING CREDENTIALS ARE ENABLED.
./scripts/release.sh 2.1.0
  • Post Release
    • Deploy artifacts with the release version
    • Verify the artifacts are on Maven Central
    • Verify the Javadocs are online with the correct version.
    • Copy the contents of bigdata/artifacts/ into the Github tag as release artifacts.


  • (LEGACY: Now done in release.sh) Create a TAG for the release in GIT. ** The TAG is the immutable snapshot of the code as of the release. Maintenance will be done on the BRANCH from which the release was issued. The TAG should use the tags path and have the general pattern .../bigdata/tags/BIGDATA_RELEASE_X_Y_Z.
git push sourceforge BIGDATA_RELEASE_X_Y_Z
git tag -a BIGDATA_RELEASE_X_Y_Z -m "Bigdata release X_Y_Z"
git push origin --tags
git push sourceforge --tags
  • Create a new directory on sourceforge for the release [2]. Upload the release notes into the directory on sourceforge and rename as "README" (they will be displayed at the bottom of the folder listing on sourceforge once you do this and refresh the page view).
  • Upload the artifacts generated by release.sh in the artifacts directory.
    • Download these artifacts and verify that the downloaded files are the same (md5/md5sum). There have been a lot of issues recently with sourceforge file uploads failing silently so you can not rely on the file being posted correctly the first time around.
    • "ant war" - generates bigdata.war (Standalone WAR deployment)
    • "ant executable jar" -- generates bigdata-X_Y_Z-bundled.jar; rename to bigdata-bundled.jar (executable jar deployment)
    • "ant deploy-artifact" - generates REL.bigdata.version.tgz (HA deployment)
    • (LEGACY) "ant javadoc" - generates javadoc. Upload to web site using rsync (see 'bigdata-project-release.txt' in dropbox for details), and verify the javadoc online.
      • (LEGACY) Rsync: rsync -qrzPl --perms --chmod=ugo+X,ugo+r -e ssh ant-build/docs www.blazegraph.com:/var/www/html/www.blazegraph.com/files/
    • (LEGACY) "ant ant-install-artifact" - generates a source code distribution. See 'bigdata-project-release.txt' in dropbox for important details on how and where to install this.
    • (LEGACY) Publish bigdata JAR to the systap maven repository:
      • NOTE: If the fastutil version has changed you will need to upload this to the maven repository. See comments in the POM.xml file.
      • (LEGACY) Change the version in pom.xml (bump version number, remove "-SNAPSHOT", do NOT commit).
      • (LEGACY) "mvn clean deploy" - deploys the new bigdata jar to the systap maven repository (alternatively try 'ant maven-deploy").
      • (LEGACY) Append "-SNAPSHOT" to the version pom.xml and commit to resume snapshot releases in CI.
    • Review the download page and edit as necessary.
    • TODO: Document process to snapshot user guide / wiki.
  • CAUTION: We maintain multiple releases. However, sourceforge will direct people to the most recently published release for which a download artifact exists. When publishing a release of a maintenance branch, you MUST (a) wait for the release to be distributed to enough mirrors that it will be assigned a download URL at which point it becomes the default download; and (b) go back and "touch" the most current release. Verify that the most current release is still the default download!!!

After actions

  • make sure that 'snapshot=true' and "#javadoc=" in build.properties and that pom.xml has -SNAPSHOT for the new version (and commit).
  • reverse merge to github (the release notes, build.properties and pom.xml are modified during the release)
    • Verify reverse merge, e.g., using Araxis Merge.
      • SF master, github master should be equal.
      • SF release tag, github release tag should be equal.
  • notify developers that the branch is open again.
  • blog it [3] and elsewhere.
  • post the release notes to the sourceforge project news (logged in as an admin, choose "News" from the tabs and then "Post New")
  • Update the website download link
  • Make sure the new files are set as the default download on Sourceforge
  • send out an email announcement (we maintain a mailing list).
  • send out a tinkerpop mailing list announcement
  • Update Wikipedia
  • update semanticweb.org (version and release date).
  • review W3 Wiki page for bigdata.
  • update the Roadmap

Maven

We publish maven artifacts for bigdata releases (starting with 1.2.3), for bigdata snapshots (from CI builds), and for bigdata dependencies that are not widely available in maven repositories. The maven repository for bigdata is:

Bigdata Maven Repository

There is a pom.xml file in the top-level of the bigdata project that controls the publication of the bigdata releases and snapshots. This pom must be updated any time a dependency version is updated or a new dependency is added. The pom also includes instructions (in XML comment blocks) on how to publish non-bigdata artifacts to the bigdata maven repository. This is used to bootstrap non-bigdata maven artifacts that are not otherwise widely available. In addition, any time we create a new release for one of the sub-projects (bigdata-ganglia, etc.), the new release artifact needs to be pushed to the maven repository.

Maven Central Guide

Maven Central Maven Deployment

PGP Key Setup

Here is a guide to moving existing PGP keys.

Encrypted Passwords

Setup your maven installation to have encrypted passwords Maven Password Encryption.