View source for External Full Text Search

= Introduction = 
In addition to its internal [[FullTextSearch]] capabilities, Blazegraph provides the means to query external fulltext indices from within SPARQL. The current implementation supports the prominent fulltext index [http://lucene.apache.org/solr/ Apache Solr] (using the HTTP Solr API), but in principle, the implementation is flexible in the sense that it allows it to hook into other fulltext search services by implementing and hooking in a simple connector class.

Note that this fulltext search feature currently is not intended to (and does not) replace the current implementation implementation of Blazegraph's internal full text index in blazegraph as decribed at [[FullTextSearch]].  The purpose of the current solr implementation rather is to support hybrid search against external resources (which may contain data going beyond what's stored in your Blazgraph instance), but not to identify matches within Blazegraph internal graph database.

= Using the External Fulltext Search Service =

To accomplish the integration of external free text search services with high-level query, Blazegraph defines several magic predicates that are given special meaning, and when encountered in a SPARQL query are interpreted as service calls to the text index.  The full list of magic predicates related to free text search is defined and documented in the class [http://www.bigdata.com/bigdata/docs/api/com/bigdata/service/FTS.html FTS]. The simplest way to integrate free text search into a SPARQL query in bigdata is to use the magic predicate '''fts:search''' inside of a SPARQL join group, plus specifying the (e.g. Solr) endpoint using the magic predicate '''fts:endpoint''' in the same join group, where the prefix fts is defined as

<pre>
PREFIX fts: <http://www.bigdata.com/rdf/fts#>
</pre>

As a simple example, consider the query

<pre>
PREFIX fts: <http://www.bigdata.com/rdf/fts#>
SELECT ?res WHERE {
  ?res fts:search "Alice" .
  ?res fts:endpoint "http://localhost:1234/solr/blazegraph/select" .
}
</pre>

In the example, the predicate <code>fts:search</code> is used to query the Solr full text index running at '''http://localhost:1234/solr/blazegraph/select''' using the search string <code>Alice</code>. Returned matches are bound as (by default) Literals to the variable appearing in the subject position, namely <code>?res</code> in the example.


== Configuring search ==

The behavior of the fulltext search can be customized through a set of magic predicates defined inside the SPARQL query (also defined in the [https://blazegraph.github.io/database/apidocs/com/bigdata/service/fts/FTS.html FTS] class). As an example, consider the following external fulltext search query:

<pre>
PREFIX fts: <http://www.bigdata.com/rdf/fts#>
SELECT ?res ?score ?snippet WHERE {
  ?res fts:search "Alice | Bob" .
  ?res fts:endpoint "http://localhost:1234/solr/blazegraph/select" .
  ?res fts:endpointType  "SOLR" .
  ?res fts:timeout "100000" .
  ?res fts:score ?score .
  ?res fts:snippet ?snippet . 
  ?res fts:params "fl=uri,description,score" .
  ?res fts:searchField "uri" .
  ?res fts:fieldToSearch "text" .
  ?res fts:scoreField "score" .
  ?res fts:snippetField "description" .
  ?res fts:searchResultType "URI" .
}
</pre>

The magic predicates have the following semantics:

* Search string and endpoint
** <code>fts:search</code>: the search query to submit
** <code>fts:endpoint</code>: the URL of the fulltext search endpoint
** <code>fts:endpointType</code>: the type of endpoint (currently, only "SOLR" is supported; this is the default anyways, so this might be skipped)
** <code>fts:timeout</code>: the timeout for the query in milliseconds 
* Additional output variables
** <code>fts:score</code>: specifies a variable to which the score of the search result is bound (requires specification of the <code>fts:scoreField</code>, see below)
** <code>fts:snippet</code>: specifies a variable to which the snippet of the search result is bound (requires specification of the <code>fts:snippetField</code>, see below)
* Search parameters
** <code>fts:params</code>: the parameter for the search, as a parameter string (URL encoded, where necessary); in the example above, we specify the fields that are returned by the Solr service, which are then used as input for the result type mapping discussed next
* Search result type mapping
** <code>fts:searchField</code>: the name of the Solr result field that is used to initialize the main output variable (i.e., the variable in subject position, namely <code>?res</code> in our example); defaults to the Solr standard field <code>id</code>, if not specified
** <code>fts:fieldToSearch</code>: the name of the Solr field that is used to search defaults to the Solr standard field <code>text</code>, if not specified
** <code>fts:scoreField</code>: the name of the Solr result field that contains the score to be returned in the variable bound through <code>fts:score</code>; note that this field must be explicitly listed as return field using <code>fts:params</code>, as illustrated by the sample query; no default, if not specified, the <code>fts:scoreField</code> variable is left unbound
** <code>fts:snippetField</code>: the name of the Solr result field that contains the snippet to be returned in the variable bound through <code>fts:snippet</code>; note that this field must be explicitly listed as a return field using <code>fts:params</code>, as illustrated by the sample query; no default, if not specified, the <code>fts:snippet</code> variable is left unbound
** <code>fts:searchResultType</code>: the type of the search result, either <code>URI</code> or <code>LITERAL</code> (which is the default); note that <code>URI</code> can only be chosen if the <code>fts:searchField</code> contains strings that can be cast to URIs (if not, a runtime exception will be thrown)

== Default Values for Magic Predicates ==
The default values for the external fulltext search magic predicates are documented at [http://www.bigdata.com/bigdata/docs/api/com/bigdata/service/FTS.html FTS], see the <code>Options.DEFAULT_*</code> members. Defaults are used whenever the magic predicate is not used; when seeing the respective magic vocabulary predicate, these default values are overridden.


== Background and Advanced Usage ==

Blazegraph's internal evaluation approach for fulltext search is to collect all basic graph patterns involving external fulltext search magic predicates, and group them into a SERVICE keyword. To give an example, Blazegraph translates a query

<pre>
PREFIX fts: <http://www.bigdata.com/rdf/fts#>
SELECT ?res ?uri WHERE {
  ?res fts:search "Alice | Bob" .
  ?res fts:endpoint "http://localhost:1234/solr/blazegraph/select" .
  ?res fts:params "fl=id,score" .
  ?res fts:scoreField "score" .
  ?res fts:score ?score .
  ?uri rdfs:label ?res .
}
</pre>

into the following query:

<pre>
PREFIX fts: <http://www.bigdata.com/rdf/fts#>
SELECT ?res ?uri WHERE {
  SERVICE <http://www.bigdata.com/rdf/fts#search> {
    ?res fts:search "Alice | Bob" .
    ?res fts:endpoint "http://localhost:1234/solr/blazegraph/select" .
    ?res fts:params "fl=id,score" .
    ?res fts:scoreField "score" .
    ?res fts:score ?score .
  }
  ?uri rdfs:label ?res .
}
</pre>


By default, the SERVICE keyword is always evaluated '''right at the beginning''' of the join group.

While in the example above the service is parameterized with constants only, in some cases it might be desirable to pass in variables that specify the behavior of the external fulltext search calls. For instance, assume you have your search terms stored as instance data in your RDF database and you want to execute a keyword search for each of them. In that case, what you may want to do is not to specify your search string as a '''fixed''' literal (namely <code>"Alice | Bob"</code>, in the example above), but pass in a variable instead, to be bound dynamically at runtime based on a given graph pattern. The following examples illustrates the idea:

<pre>
PREFIX : <http://my.namespace/>
PREFIX fts: <http://www.bigdata.com/rdf/fts#>
SELECT ?searchTerm ?res WHERE {

  SERVICE <http://www.bigdata.com/rdf/fts#search> {
    ?res fts:search ?searchString .
    ?res fts:endpoint "http://localhost:1234/solr/blazegraph/select" .
    ?res fts:params "fl=id,score" .
    ?res fts:scoreField "score" .
    ?res fts:score ?score .
  }
  ?searchTerm rdf:type :SearchTerm .
  ?searchTerm :searchString ?searchString .
}
</pre>

In the examples, we assume that the triple pattern <code>?searchTerm rdf:type :SearchTerm . ?searchTerm :searchString ?searchString</code> binds variable <code>?searchString</code> to a set of search terms. Note that Blazegraph applies some analysis of the dependencies between variables and delays the execution of the service call until all required variables have been bound. In that case, this means that the triple patterns <code>?searchTerm rdf:type :SearchTerm . ?searchTerm :searchString ?searchString .</code> are executed prior to the service call.

An alternative approach (or for even more complex examples), you may also consider using the <code>WITH</code> Blazegraph extension (see [[NamedSubquery|here]]), which allows you to bring subqueries in any order you want, which gives you full control over them.

== Pitfalls ==
If used inproperly, the use of external fulltext search might result in runtime exceptions. This might happen particularly if

* the search string is not bound or empty
* the specified fulltext search endpoint is not provided, empty, or incorrect
* the <code>fts:searchResultType</code> is set to <code>URI</code>, but the specified <code>fts:searchField</code> contains strings that cannot be converted to URI
* the query <code>fts:timeout</code> is exceeded

In such cases, the exception message should give pointers to the problem.


= For Developers: Implementing Other External Fulltext Services =
The <code>fts:endpointType</code> predicate specifies the type of the endpoint. Currently, only Solr endpoints are supported (<code>fts:endpointType</code> equals "SOLR"), but this is the switch that you could use to implement connectors to other types of endpoints. To do so, the following needs to be done:

* Fix a new endpoint type (say, e.g., "ElasticSearch") and extend enum [http://www.bigdata.com/bigdata/docs/api/com/bigdata/service/fts/FTS/EndpointType EndpointType]
* Implement a class, say <code>ElasticFulltextSearchImpl</code> that extends the IFulltextSearch<FulltextSearchHit> Method

<pre>
@Override
public FulltextSearchHiterator<FulltextSearchHit> search(
       com.bigdata.service.fts.IFulltextSearch.FulltextSearchQuery query) {

   // your code goes here 

}
</pre>

receives an in-memory object of the fulltext search query (i.e., an in-memory representation of what is specified in the SPARQL query) as input and must output an iterator over search hits. You may want to take a look at  [http://www.bigdata.com/bigdata/docs/api/com/bigdata/service/fts/impl/SolrFulltextSearchImpl SolrFulltextSearchImpl] as the reference implementation for Solr.

* In class <code>FulltextSearchServiceFactory</code>, extend the switch over the <code>endpointType</code>, resolving the new enum value to your new implementation