Scale OutCI

From Blazegraph
Jump to: navigation, search

This page provides guidelines for running various unit tests against a bigdata federation. The basic procedure involves the setup of a 2-DS (data service) federation running on a single node. This setup is sufficient to exercise most aspects of scale-out, including service discovery, RMI, NIO, and cross data service sharded index operations. However, note that index sharding is driven by the data scale so it is not really possible to exercise a bigdata federation at small data scales used by the unit tests.

Setup 2-DS federation on one node

The following procedure may be used to start a federation on a single node. This assumes that you have the source code checked out in the REL directory. You may need to edit the FED, NAS, LAS, REL, and DIST variables for your environment.

FED=benchmark
NAS=/nas/bigdata/${FED}
LAS=/data/bigdata/${FED}

# Source directory for release version.
REL=~/workspace/BIGDATA_RELEASE_1_3_0

# Distribution specific version.
DIST=bigdata-1.3.1

# create the DIST
cd $REL
ant ant-install-artifact

# extract DIST to DIST specific directory
rm -rf /nas/bigdata/${DIST}
tar xzvof DIST.${DIST}.tgz -C /nas/bigdata/

cd /nas/bigdata/${DIST}

# remove the existing install ($LAS is cleared by bigdata destroy)
rm -rf $NAS

# install distribution.
ant ant-install

# setup environment
source $NAS/bin/bigdataenv

##
# Replace any edits to the bigdataCluster config file before starting the cluster.
# 
# See $NAS/config/bigdataCluster.
##
bigdata start

# Note: Typically, this is done in a second terminal window so you can view error messages during startup, if any.
tail -f $errorLog

Run test suites

SPARQL Test Suite

The class com.bigdata.rdf.sail.tck.BigdataFederationSparqlTest may be used to run the SPARQL test suite. Instructions for running this test are in the javadoc at the top of the class. Running the class looks like this:

java -cp ... -server -Djava.security.policy=policy.all -Dbigdata.configuration=/nas/bigdata/benchmark/config/bigdataStandalone.config com.bigdata.rdf.sail.tck.BigdataFederationSparqlTest

NanoSparqlServer Test Suite

The class com.bigdata.rdf.sail.webapp.TestNanoSparqlServerWithProxyIndexManager may be used to run the NanoSparqlServer test suite. Instructions for running this test are in the javadoc on the main() routine in the class. Running the class looks like this:

java -cp ... -Djava.security.policy=policy.all com.bigdata.rdf.sail.webapp.TestNanoSparqlServerWithProxyIndexManager triples /nas/bigdata/benchmark/config/bigdataStandalone.config

The test suite SHOULD be run once for each of the inputs "triples", "sids", and "quads". This verifies the behavior of the NanoSparqlServer against the three primary RDF database modes.

Tear down the federation

WARNING : Depending on how you configured the top-level build.properties (e.g., forceKillAll=true), the following MAY issue killall -9 Java. To be safe, run the scale-out CI tests on a node that is not running any other JVM processes.

bigdata destroy