Release 4.4.0.1

This document lists all changes for the most recent major release (releases with number X.Y) and subsequent minor releases (with numbers X.Y.Z and perhaps further identifiers), including major and important changes and a list of all user-visible modifications. Change History repeats the list of user-visible modifications for this release and includes similar lists for earlier releases.

The release described in the documentation set is 4.4.0.1.

AllegroGraph 4.4.0.1 is a maintenance release with minor fixes and performance enhancements but no major new features. The changes in the previous major release, version 4.4, are listed just below followed by changes in the previous major release.

The agraph-backup program shipped with 4.4 and 4.4.0.1 had an issue that per-store namespace settings weren't carried over. This is corrected in an update available for download. The new version of agraph-backup can be downloaded from http://franz.com/ftp/pri/acl/ag/ag4401/linuxamd64.64/SE/agraph-4.4.0.1-backup-20120112.tar.gz . The user interface is changed and the documentation now describes the new interface. See Database Upgrade.

Major Changes in release 4.4

• Non backward-compatible change: duplicate deletion is no longer done in the background by choosing a deletion strategy. Instead, duplicates are only deleted when delete-duplicate-triples is called (either a direct call to the Lisp function or using one of the APIs).

• Support for SPIN. The SPIN API allows you to define a function in terms of a SPARQL query and then call that function in other SPARQL queries. These SPIN functions can appear in FILTERs and can also be used to compute values in assignment and select expressions. See Spin.

• agraph-backup now can preserves settings as well as data and can be used when upgrading to a new version. See Backup and Restore and Database Upgrade. convertdb is still supported but deprecated.

• The agraph-java-client library now has a feature for pooling connections to the AllegroGraph server. See Java Connection Pooling.

• All features of SPARQL 1.1 are supported. See the SPARQL 1.1 Release Notes for more details.

• A new version of agload has more features and a new interface. Old calls work except you can no longer specify a directory as a value of the FILE argument to have all files in the directory loaded. See Data Loading.

• There have been numerous internal improvements, particularly to index updating and query processing, which should result in improved performance.

Changes in release 4.3.x

The major changes in that various 4.3.x releases included the following. See the Change History for a list of all changes in the various 4.3.x releases (and for changes in earlier releases).

• Triple-level security: you can now block triples with specified values as the subject, predicate, object, or graph from being seen or modified by individual users or all users with a specified role. See this section in Security Implementation for details.

• Remote API works on many more machines: in earlier relases, you had to run the remote API on a 64-bit Linux machine or a simulator of one. Now Windows machines and Macs may be used, as well as 32-bit Linux machines. See Lisp Quickstart.

• Stored procedures: these have been supported but not documented for a while. Now they are documented. See Stored Procedures.

• Full text indexing supports Japanese text. See create-freetext-index.

AllegroGraph 4.4.0.1 user-visible changes

AllegroGraph Server: General Changes

Rfe11168 - Improvements to query statistics computation and usage

Made several incremental improvements to the statistics used to determine query ordering. This increases the performance of some complex queries.

Rfe11168 - Improvements to query statistics computation and usage

Made several incremental improvements to the statistics used to determine query ordering. This increases the performance of some complex queries.

Bug20667 - XSD float exponent fix

Due to rounding issues when encoding, exponents of floats in the :sparql-xml result format were - very seldom - off by one. This has been fixed.

Bug20615 - TrxLogRetain replication now works on secondaries

The catalog directive: TransactionLogRetain replication now archives properly for replication secondaries.

HTTP Client

No significant changes.

SPARQL

Rfe11219 - Add query option support for all SPARQL constructs

AllegroGraph's SPARQL parser supports specifying query options as part of the query string. These take the form

prefix franzOption_OPTIONNAME: <franz:OPTIONVALUE> 

In particular, you can tell AllegroGraph which query engine to use with the queryEngine parameter. For example, to use the SPARQL 1.0 engine in a query, you would add the prefix

prefix franzOption_queryEngine: <franz:sparql-1.0> 

This extension is now supported for all SPARQL constructs: ASK, CONSTRUCT, DESCRIBE, SELECT and UPDATE.

Rfe11218 - Improve maximum solution size handling

Because the SPARQL 1.1 engine operates on the entire query solution simultaneously it can be more memory intensive than the SPARQL 1.0 engine for some queries. There was an internal check to warn if query solution sets were becoming too large. This check was causing some queries to fail rather than simply signaling a warning. This has been changed so that these queries will write a warning to the log rather than failing.

Rfe11202: SPARQL Update example with batch update

With this change, Java tutorial example 13 has been augmented to show how to evaluate SPARQL Update queries, including one that batches updates in a single request. [This item is repeated in the Java Client section.]

Rfe11070 - improve performance of variable predicate queries

AllegroGraph's SPARQL engine now recognizes queries that have variables in the predicate position and provides custom methods to optimize them. An example of such a query is:

SELECT ... {  
  ... clauses ...  
  ?s ?p ?o . 

The optimizations are only possible when AllegroGraph has already computed the complete list of predicates in the store. This can be accomplished using the ComputeStatistics query option. For example

PREFIX franzOption_ComputeStatistics: <franz:yes>  
SELECT ... 

Bug20768 - SPARQL 1.1 property paths could return incorrect results

AllegroGraph was failing to filter non-matching rows for property path queries like

select ?a {  
  ?a owl:sameAs{1} ?a .  
} 

meaning that it would emit results when it should not. This has been fixed.

Bug20692 - Sparql 1.1 CONTAINS function can fail

Corrected a problem with some instances of the SPARQL 1.1 CONTAINS function.

Bug20676 - Geospatial datatype mappings could cause query errors

Changes to some query optimization code in the AllegroGraph SPARQL engines (both sparql-1.0 and sparql-1.1) were incompatible with geospatial datatype mappings (geospatial predicate mappings were not effected). This could lead to parse errors during query processing. This has been fixed.

Bug20665 - SPARQL-1.1 UNION doesn't see with-variable bindings on the RHS

The SPARQL 1.1 engine was losing track of the variable bindings on the right-hand side of a UNION. This could cause queries like

select ?s1 ?s2 {  
  { ?s1 a ?p . }  
  union  
  { ?s2 a ?p . }  
} 

(with a variable binding for ?p) to produce incorrect results. This has been corrected.

Bug20664 - SPARQL-1.1 CONCAT function signaling errors with typed literals

The SPARQL-1.1 CONCAT function could signal an error instead of generating the correct result when one or both of the arguments with a typed literal. This has been corrected.

AGWebView

Rfe11198 - Improve WebView's bulk loading UI

WebView no longer conflates bulk load mode with agload usage. It's now possible to use agload's multi-core optimized loading mechanism and to optionally turn bulk load mode on. Some aspects of the file loading UI in WebView have also been optimized: The format selection element is now a combo box, and is located closer to the file input field.

Bug20662 - Reload triple count in WebView if load fails

WebView will now display the correct triple count in a store after an import got stopped by an error (e.g. the triple license limit).

Bug20641 - Improve restoring backups from WebView

When restoring from a backup in the WebView catalog overview, AllegroGraph will now automatically upgrade the newly-created store to the newest version. Additionally, there is now an option to set the store up for use as a replication secondary.

Bug20637 - Fix problem where SPARQL Update did not work with AGWebView

Changes to SPARQL UPDATE caused some UPDATE queries to fail in AGWebView. This has been corrected.

Changes to the Lisp API

Bug20640 - Fix printing of encoded-ids in remote lisp client

AllegroGraph's remote lisp client now correctly translates encoded IDs' UPIs back into strings. It can also correctly translate variable-width encoded IDs into UPIs now.

Documentation

No significant changes.

Python Client

No significant changes.

Java Client

Rfe11202: SPARQL Update example with batch update

With this change, Java tutorial example 13 has been augmented to show how to evaluate SPARQL Update queries, including one that batches updates in a single request. [This item is repeated in the SPARQL section.]

AllegroGraph 4.4 user-visible changes

AllegroGraph Server: General Changes

Rfe11082: Transactional duplicate deletion

INCOMPATIBLE CHANGE: AllegroGraph no longer supports duplicate deletion at index optimization time. The database creation parameters indicating duplicate deletion strategy are no longer accepted.

Instead, AllegroGraph now allows deletion of duplicates using the delete-duplicate-triples API function, which is also accessible via the HTTP API call DELETE /statements/duplicates.

Rfe11039 - Include AllegroGraph version in ps output

AllegroGraph subprocesses modify their command line so that they can supply identifying information to programs like 'ps' and 'top'. Now, in addition to the name of the subprogram, the version of AllegroGraph is included. Also, for the AllegroGraph Service Daemon process, the port number that it is listening on is included.

Rfe11017 - Add an agdebug command line utility

AllegroGraph now ships with a command line utility called agdebug that will output a summary the contents of the transaction log files in a store. This is primarily useful for diagnosing problems with transactions and the database's storage system.

Rfe10984 - Fix serialization issues with encoded IDs

Both the turtle and the RDF/XML serializers failed to correctly handle encoded IDs. This has been corrected.

Rfe10981 - Improve upgrading/migration process

The agraph-backup tool has two new commands, backup-all and restore-all, to help migrating existing databases to the new version of AllegroGraph or copying all catalogs and repositories to a new location. See Backup and Restore and Database Upgrade.

Rfe10948 - Log secondary index creation

Adds log entries to the AllegroGraph log when a secondary index (also called a UPI map) is created.

Rfe10853 - Support fti charset in Python client

Keyword arguments innerChars, borderChars, and tokenizer have been added to createFreeTextIndex and modifyFreeTextIndex.

getFreeTextIndexConfiguration also returns these new configuration properties and values. See put-freetext-index.

Rfe10783 - Add per triple-store control of secondary indices

Modified the Lisp and HTTP APIs so that each triple-store can specify whether or not it should use existing secondary indices and/or create new secondardy indices during query. Since these indices must be maintained and updated when modifying the triple-store, automatic creation should be turned off until the store is quiescent. The default is to use secondary indices when they exist but not to create new ones. Use [create-secondary-indices-p[]/(setf create-secondary-indices-p), and use-secondary-indices-p/(setf use-secondary-indicies-p)* to control the behavior.

Rfe10568, Rfe11172 - Set the default query engine via agraph.cfg

AllegroGraph supports two SPARQL query engines. The algebra engine has been available for several years and is well tested, stable and robust. The newer, set-based, engine, named sparql-1.1 can be significantly faster than the algebra engine but may contain bugs that the algebra engine does not.

Out of the box, AllegroGraph uses the sparql-1.1 engine. This can be changed by using the QueryEngine parameter in the configuration file. If you include

QueryEngine algebra 

Then AllegroGraph will use the algebra engine. If you leave the setting off or use

QueryEngine sparql-1.1 

Then AllegroGraph will continue to use the set based query engine.

The value of this configuration is read each time a triple-store is created or opened.

Rfe10179, Rfe10540 - SNA optimizations

Improved the efficiency of SNA operations in several ways. All SNA generator functions are now slightly faster. SNA generators that use select now save the select plan and re-use it. This can make these generators hundreds of times faster than before. The SNA system will also use secondary indices (UPI-maps) when available.

Rfe10398 - Improved speed of low-level AllegroGraph operations

Optimized the code used to copy UPIs and parts of triples. This produces a query speedup in SPARQL queries and most Prolog select queries. Some count-only Prolog select queries can run slightly slower.

Rfe10090, Rfe10541 - optimizations to AllegroGraph secondary indices

Improves both the creation of secondary indices (UPI-maps) and their efficiency in use. Queries that make heavy use of secondary indices will operate more quickly.

Rfe9786 - Make shm server global

A new configuration directive GlobalShmServerDirectory is added to specify a directory where the global shared-memory server creates its socket. When omitted, the value of SettingsDirectory is used.

Bug20619 Optionally not inherit trx log arch params during restore

The configuration file catalog directive TransactionLogArchive is no longer inherited from a backup database during a restore. In addition, when the --replica flag is set during agraph-backup restore, TransactionLogRetain and TransactionLogReplicationJobname are no longer inherited from the backup and must be set manually in the config file catalog section prior to restore. This is so that a replication primary and replication secondary can have independent transaction log archiving behavior. The flag --replica also ensures --nocommit is set and --newuuid is not set, and should be used as a replacement for --nocommit when restoring backups for replication client purposes.

Bug20460 - Encoded IDs and reasoning triple-stores mix not

If reasoning was used during a query, then encoded IDs were not correctly recognized which led to incorrect results. This has been corrected.

Bug20443 - Encoded IDs did not work with the TriX serializer

The TriX serializer was incorrectly emitting encoded IDs as if they were plain literals instead of resources. This has been corrected.

Bug20355 - Durably save transactions on warm-standby client

The warm-standby secondary machine now saves transaction log entries in a way that ensures they've been flushed to disk durably as soon as the entire commit has been read.

Bug20332 - Trix parser can get confused and try to encode plain literals

Under some circumstances, the TriX parser could attempt to encode plain literals as if they were typed literals which could cause a data import to fail. This has been corrected.

Bug20230 - Fix deletion slowdown under replication

The replication server used a connection to the replicated triple store that ensured it would never expunge deleted triples, slowing down operations if many deletes had occurred. Now, the replication server correctly expunges these triples and does not slow down.

Bug19957 - Turtle parser should support type-mapping

The turtle parser was not respecting a triple-store's type-mapping settings. This meant that triples added via turtle would not have their objects converted into encoded UPIs. This has been corrected.

Spr38044 - Restrict possible values for TransactionLogRetain

AllegroGraph will now signal an error when invalid values for TransactionLogRetain are encountered in the config file.

HTTP Client

No significant changes.

SPARQL

Rfe10985 - Spin functions and magic-properties

SPIN functions and magic-properties have been implemented. The SPIN implementation is described in Spin document.

Rfe10737 - SPARQL 1.1 BIND support

Add support for SPARQL 1.1's BIND language construct. In particular, queries like

select ?title ?costPlusTax {  
?book <http://purl.org/dc/elements/1.1/title> ?title .  
?book <http://example.org/ns#price> ?cost .  
?book <http://example.org/ns#tax> ?tax .  
bind( ?cost * (1 + ?tax) as ?costPlusTax )  
} 

are now supported.

Rfe10736, Rfe10743 - SPARQL 1.1. exists, not exists, and minus

AllegroGraph's SPARQL engine now handles the MINUS operator of SPARQL 1.1. It also handles the new EXISTS and NOT EXISTS filter operators.

Rfe10700 - SPARQL 1.1 assignment

SPARQL 1.1 assignment is now implemented so that queries like

select ?title ?cost ?discount (?cost * ?discount as ?totalCost)  
where {  
?book :title ?title .  
?book :price ?cost .  
?book :discount ?discount .  
} 

are now supported.

Rfe10699 - SPARQL 1.1 UPDATE support

The SPARQL parser and implementation now support the UPDATE specification at http://www.w3.org/TR/2011/WD-sparql11-update-20110512/.

Note: This change is incompatible with some existing uses of SPARQL UPDATE.

AGWebView

Rfe11131: Allow triggering deletion in WebView

It is now possible to invoke duplicate deletion from WebView.

Rfe10631 - Allow loading files via agload in WebView

WebView now has a "Bulk Load" option to speed up loading large files. Since this option uses the agload utility internally, it also allows specifying an error handling strategy other than "Cancel".

Bug20492 - Improve code editor in WebView

The text editor in WebView (used for e.g., queries) has been improved for browser compatibility (mainly with Internet Explorer) and functionality. It now expands automatically to fit the text contained, displays line numbers and automatically indents code.

Changes to the Lisp API

Rfe11082: Transactional duplicate deletion (repeated from the General section)

INCOMPATIBLE CHANGE: AllegroGraph no longer supports duplicate deletion at index optimization time. The database creation parameters indicating duplicate deletion strategy are no longer accepted.

Instead, AllegroGraph now allows deletion of duplicates using the delete-duplicate-triples API function, which is also accessible via the HTTP API call DELETE /statements/duplicates.

Documentation

No significant changes other than new features documented.

Python Client

Rfe11122 - Add "delete duplicates" API function to python client

RepositoryConnection.deleteDuplicates:

Delete all duplicates in the store. Must commit.

mode - "spog" or "spo"

Java Client

Rfe11121 - Support transactional duplicate deletion

With this change, the Java client now provides the method AGRepositoryConnection#deleteDuplicates(comparisonMode) for deleting spog and spo duplicates in a transactional manner. See the javadoc for more details.

See Change History for user-visible changes to earlier releases.

Rfe11117: java api for SPIN (adds to rfe10988)

New methods added to AGRepositoryConnection: listSpinFunctions() listSpinMagicProperties()

New classes: AGSpinFunction, AGSpinMagicProperty

Rfe10622 - Http response handlers now have their own package

With this change, the Java client's HTTP response handlers have been refactored and moved into a com.franz.agraph.http.handler package.