• Introduction
• Metaobject Protocol for Java
  • Transparent persistency and concurrency control
  • JavaMOP utility
  • Basic metaobjects
• How to develop Java application for GOODS
• Using Interface types and the PersistenceFactory
• Per-thread transaction
• Raw binary types
• Persistent class library
  • Dynamic arrays
  • Ordered and unordered sets
  • B-Tree
  • Blob
  • Closure
  • Hash table
  • H-Tree
  • Class library
  • Java embedded SQL
• Compatibility with C++
• Building GOODS Java client library
• Native socket library
• JavaDOC documentation
• Examples
  • Guess an animal
  • Graphic editor
  • Test of Blob
  • Test of B-Tree

Introduction

Currently interfaces for the C++ and the Java language are provided. The C++ interface is based on using smart pointers. It is discussed in the "readme.htm" document. The interface for the Java language is based on using a special Java byte-code preprocessor. It is described in the following sections.

Metaobject Protocol for Java

Transparent persistency and concurrency control

One of the desired features of an OODBMS programming language interface is the support of orthogonal persistency. This means that the property of the object to be persistent or transient is orthogonal: independent from the class and the constructor used for object creation. This purpose can be achieved through persistency by reachability. The object becomes persistent when it is reachable (when there exist references to the object) from persistent roots or from other persistent objects. But complete support for orthogonal persistency causes significant runtime overhead. That is why the GOODS language interface provides the semi-orthogonal approach: only objects of some selected classes (they should be derived from special base classes) can become persistent. We will call them persistent capable classes. The principle of persistency by reachability is still used here: not all objects of the persistent capable class become persistent, but only those which are reachable from persistent roots. Such approach allows not to pay a performance penalty for normal (transient) objects, and provides efficiency of the OODBMS language interface implementation.

JavaMOP utility

It was decided to use a byte-code preprocessor. The development of such a preprocessor is much simpler than the development of a source-level preprocessor (no language parser must be constructed). Also the absence in the Java language of such constructions like the C "#line" directive, makes compiling and debugging applications with a source-level preprocessor very inconvenient. That is why a byte-code preprocessor JavaMOP was developed. It is implemented in C++, to increase the speed of processing and makes the development more comfortable.

JavaMOP receives as a parameter a list of Java classes and/or directories or archives of Java classes (.JAR or .ZIP without compression). If an archive file is specified, JavaMOP is able only to read the definition of classes in this archive, but is unable to preprocess these classes. To preprocess some class library, you should first extract all files from the archive and pass the root directory of the extracted tree to JavaMOP. If the directory name is passed, JavaMOP will handle all .class files in the directory and recurse into all subdirectories.

JavaMOP looks for classes containing a "metaobject" field as an instance variable. All such classes and classes derived from them are considered to be controlled by metaobjects. JavaMOP wraps bodies of methods of all such classes with calls of the metaobject methods preDaemon() and postDaemon(). As it is clear from their names, the first is invoked before execution of the first method statement; the last is invoked after returning - normally or as a result of exception - from the method. JavaMOP also detects accesses to non-self instance variables of such classes. JavaMOP wraps such accesses in all methods (not only in instance methods of objects controlled by metaobjects) with invocations of preDaemon() and postDaemon() methods.

To implement a transparent access to the database, it is necessary to detect the moment when the object is modified; and distinguish methods, which access the object in read-only mode, from methods, which can modify the object. Instance methods, which can (but not necessarily do) modify self objects, are called in GOODS mutators. Knowledge of whether a method is a mutator or not is very significant to select the proper object locking mode. The naive approach is to lock an object in shared mode when it is first accessed; and to upgrade the lock to exclusive mode, when it is going to be modified. But such approach is deadlock prone. Consider two applications accessing the same object. Both set a shared lock and then try to modify the object. But none of them can grant exclusive lock since other application has shared lock, preventing from locking object in exclusive mode. But none of them can grant an exclusive lock since the other application shares the lock, preventing both from locking the object in exclusive mode.

JavaMOP uses a two-pass algorithm to correctly calculate the set of all mutator methods. All methods modifying their object instance variables are considered to be mutators. If some method f() invokes method g() for the same object, and method g() was marked as a mutator, then method f() should be also marked as a mutator. The situation becomes more complicated because of polymorphic calls (virtual methods). We should mark method f() as a mutator if any of the implementations of method g() in the self class or a derived class was marked as a mutator.

During the first pass, JavaMOP generates a methods call graph and marks as mutators all methods with instructions assigning some values to self instance components. Then JavaMOP propagates mutator attributes, iteratively scanning the methods call graph and marking as a mutator each method, which invokes some mutator method. After that, code is generated for all methods with known status of each method.

To detect the moment of object modification, JavaMOP allocates an extra local variable and assign to it a non-zero value before each store instruction to a self instance field. When the postDaemon() metaobject method is called, the value of this variable is passed to it to determine whether the object was modified by the method. JavaMOP reduces the size of generated code by doing basic block optimization: if there are several store instructions within one basic block, the modification marker variable can be assigned only once.

Constructors should be handled in a special way since it is not possible to wrap a constructor body with invocations of metaobject methods (because the first statement of a constructor should be the invocation of the constructor of the superclass). So JavaMOP inserts a call of the postDaemon() method only after the top-level construction invocation statement, following the object creation by the new operator. The method preDaemon() can be called by applications explicitly from the constructor of some base class.

Although JavaMOP will do its best to set the mutator and the modified attributes, it is possible for an application to explicitly mark some method as a mutator or an object as modified. This can be done by invocating two special static methods: mutator() and modify(). JavaMOP removes code invoking these methods and, instead of this, changes the correspondent attributes. So a metaobject protocol, provided by JavaMOP, is very simple and consists only of two primary methods:

public abstract class Metaobject { 
    //
    // Invocation of this method is inserted by MOP generator before each
    // method invocation or object component access.
    //
    abstract public void preDaemon(Object obj, int attr);
    //
    // Invocation of this method is inserted by MOP generator after each
    // method invocation or object component access.
    //
    abstract public void postDaemon(Object obj, int attr, boolean modified);


    //
    // Attributes for daemons 
    //
    final static int MUTATOR    = 1; // object can be changed
    final static int VARIABLE   = 2; // access to non-self instance variable 
    final static int CONSTRUCTOR= 4; // wrapped method is constructor
    final static int EXCEPTION  = 8; // method is terminated by exception 

    //
    // This method is only hint to the MOP preprocessor to consider a method
    // invoking Metaobject.mutator() as beeing a mutator. No actual code
    // will appear in the preprocessed class file.
    //
    public static void mutator() {} 

    // 
    // This method provides the  MOP preprocessor with the information that objects
    // were (or will be ) modified. No method invocation will appear in the
    // preprocessed code.
    //
    public static void modify() {} 
};

JavaMOP provides some other useful facilities. For each class, controlled by a metaobject, it generates a special constructor with a single Metaobject parameter, which is passed to the constructor of the base class. Such constructor is used in GOODS to create an object instance when it is loaded from the database. But using this generated constructor makes it impossible to declare final instance components in classes controlled by metaobjects (JavaMOP is not able to insert correct initialization code for such components in the generated constructor, causing a verification error). The attribute final can not be used even with instance fields declared as transient.

The second JavaMOP feature can be used to overcome the limitations of the reflection facilities of JDK 1.1, which violates access to non-public components of non-public objects (in JDK 1.2 it is possible to set a bypass flag and access all components of all objects). As far as there is only a beta release of JDK 1.2 available at this moment (and even after JDK 1.2 will be available, a lot of users will continue to use JDK 1.1), JavaMOP has a special option -public, which tells JavaMOP to set the public attribute to all MOP classes and their fields. Using this option is not needed with JDK 1.2.

The JavaMOP option "-classes" allows you to specify a list of classes, which should be controlled by metaobjects. In this case exactly one class with a "metaobject" instance field should be provided to JavaMOP. We will call this class the MOP root class. JavaMOP will transform the definition of the classes, specified in the list, to make them be derived from the MOP root class. The option "-classes" should be followed by a file name. This file should contain a list of full class names, separated by space characters (new line, space, tabulation...). The class name should be written in the same format as in the Java import statement. For example: "goodslib.ArrayOfChar" or "goodslib.*". If the '*' symbol is placed at the end of a class name, then all classes from the specified package and nested packages are implicitly derived from the MOP root class. Classes specified in the "-classes" option list should not be archived, otherwise JavaMOP will not be able to preprocess them.

When JavaMOP is used for the GOODS client interface, the class goodsjpi.Persistent, containing a metaobject instance field, is available. All persistent capable classes should be derived from this Persistent class (it can be done implicitly by JavaMOP, see paragraph below). So the set of persistent capable classes is equal to the set of MOP classes (i.e. the classes controlled by metaobjects).

It is possible to use the existing Java class library without changing sources by putting the list of persistent capable classes in some file and using JavaMOP with the "-classes" option. The following rules should be observed carefully when using this option:

1. A class should not have more than one instance array component.
2. If the instance array component is modified (by storing array elements), the method Metaobject.modify() should be called explicitly to mark the object as modified.
3. All classes, which objects can be referenced from some other persistent capable class, should also be persistent capable (so persistent capable classes should form closed sets in the sense of interclass references).

It is significant to notice that although JavaMOP was designed for the GOODS Java interface, it is not GOODS specific and can be used in any application requiring non-standard manipulation with object instances.

JavaMOP performs the following byte code transformations:

1. Add prologue/epilogue to each method of persistent capable object For example, method

   void foo() { 
           // do something
   } 
   

   with be transformed to

   void foo() { 
           metaobject.preDaemon(this, 0);
           try { 
                   // do something
                   metaobject.postDaemion(this, 0);
           } catch (Throwable x) { 
                   metaobject.postDaemion(this, Metaobject.EXCEPTION);
           }
   }
   

2. All accesses to the not-self instance variable of persistent capable class are wrapped with metaobject method:

   void func(PObject po) { 
           po.x = 0;
   }
   

   will be transformed to

   void func(PObject po) { 
           po.metaobject.preDaemon(po, Metaobject.MUTATOR|Metaobject.VARIABLE);
           po.x = 0;
           po.metaobject.postDaemon(po, 0);
   }
   

3. Invocation of constructor of persistent capable object is ended with postDaemon call:

   PObject po = new PObject();
   

   will be transformed to

   PObject po = new PObject();
   po.metaobject.postDaemon(po, Metaobject.CONSTRUCTOR);
   

When it is possible to avoid preprocessing of class files with JavaMOP? Lets say we have two classes PObject, which is persistent capable class, and TObject which is not persistent capable class (transient class):

class PObject extends Persistent { 
        PObject next;
        int     x, y;

        void f(TObject to) { 
                to.g(this);
        }
}

class TObject { 
        void  g(PObject po) {
                // do something with po object but do not modify it  
        }
}

But if the program contains code like this

class X {
        void foo() { 
                PObject po = new PObject();
                TObject to = new TObject();
                to.g(po);
        }
}

class TObject { 
        void  g(PObject po) {
                PObject anotherPO = new PObject();
                ...
        }
}

class TObject { 
        void  g(PObject po) {
                while (po != null) { 
                        po.f(this);
                        po = po.next;
                }
                ...
        }
}

You can specify all files which should be preprocessed by JavaMOP in command line (may be using wildcards, for example *.class), or you can specify only the name root directory (or names of several directories) which will be recursively traversed by JavaMOP. Only the files with *.class or *.CLASS extension will be preprocessed by JavaMOP while traversing the tree.

If you want to create archives with persistent capable classes (ZIP or JAR), they should be build without compression (-0 flag for zip or jar) in order to let JavaMOP to read this files. JavaMOP currently is not able to unpack compressed ZIP files. JavaMOP will not patch files extracted from archive. Also JavaMOP adds special string to the preprocessed class file, so it will not patch it next time (in other words it is possible to apply JavaMOP several times to the same file - result will not be changed). JavaMOP doesn't use CLASSPATH and process only the classes specified in command line. Moreover, as far as JavaMOP should build persistent close to perform correct bytecode transformation (i.e. detect set of all classes which are persistent capable), it is necessary to specify ALL class files which are persistent capable OR access persistent capable objects (so if you apply JavaMOP idividually for each class file, then produced code will not be correct)

Basic metaobjects

Most of the metaobject functionality is implemented in the BasicMetaobject class. This class is responsible for managing the object cache, controlling the transaction processing and handling the object invalidation messages from the servers. GOODS provides an implicit model for fixing transactions: each invocation of the method of an object controlled by the BasicMetaobject is considered as a nested subtransaction. The transaction is automatically committed when all nested subtransactions are terminated (in other words, when there are no more active methods of classes, which are controlled by a metaobject). The BasicMetaobject metaobject invokes the beginReadAccess(), beginWriteAccess() and endAccess() methods, which implement the concrete synchronization policy by setting shared or exclusive locks. Three policies are supported now and so three metaobjects, derived from the BasicMetaobejct, are defined:

OptimisticMetaobject

Accessed objects are not locked at all. When the transaction is committed, the server checks if the client has the most recent version of the object. If some other client modified this object before this client, then the transaction is aborted by the server. It depends on the application how to handle this situation. Objects, which are only read and not modified by the application during the transaction, are not checked by the server to be up-to-date. This strategy is efficient, when conflicts are rare and it is possible to restart the transaction. It also can be used when the object is always accessed through some other object (container), which is responsible for synchronization.

PessimisticMetaobject

This metaobject sets an exclusive lock on the object, when it is accessed by the mutator method, to prevent loss of modifications as a result of a transaction abort. Locks are released at the end of the transaction. This metaobject doesn't lock objects accessed in read-only mode; such objects are not checked by the server to be up-to-date.

PessimisticRepeatableReadMetaobject

This metaobject is derived from the PessimisticMetaobject. In addition to setting exclusive locks for modified objects, it also locks read-only objects in shared mode to prevent the object from modification by other clients until the end of the transaction. This metaobject is used by default for all persistent capable classes. It provides the highest level of consistency, but at the expense of extra message passing, thus reducing concurrency. Also it can cause deadlocks by upgrading shared locks to exclusive locks.

The basic metaobject controls the work of the object cache. Since loading objects is the most frequent operation in a OODBMS, the performance of a database application mostly depends on an effective cache mechanism. Therefore special attention was paid to the implementation of the cache object replacement discipline in the GOODS client library. The most popular discipline for cache management is LRU (replacing the least recently used object). But it is not smart enough for database applications. For example, a sequential search through a large number of objects can completely flush the cache by replacing all objects in it although most of the replacing objects will not be used by the application in the future.

To solve this problem, an extension of the standard LRU scheme was proposed in GOODS. The object cache is divided into two parts: a primary cache and a cache for frequently used objects.The first time the object is accessed, it is placed in the primary cache. The object will be moved to the "frequently used objects" cache only when it is accessed several times. Both parts of the cache are managed by a standard LRU discipline separately: when the total size of objects in one part exceeds some limit, all least recently used objects from this part are thrown away from the clients memory, not affecting objects in the other part of the cache. It is possible to set limiting values for both parts of the cache by the BasicMetaobject.setCacheLimits(int l0, int l1) method. The parameter l0 specifies the limit for the primary cache, and l1 for the cache of the frequently used objects.

How to develop Java application for GOODS

Special conventions apply to array types as components of persistent capable objects. Since Java arrays are derived only from the Object class, they are not persistent capable. Instead of this, GOODS interprets a persistent object with a single array component as an object with varying length (corresponding to the C++ type with varying length of the last component: struct string { int length; char body[1]; }; ). So it is not possible to have more than one array component in a persistent capable class. Furthermore you have to explicitly use the Metaobject.modify() and/or Metaobject.mutator methods, because JavaMOP is not able to detect modifications of array components. It is better to avoid using Java arrays for components of persistent objects. Instead use the dynamic array classes from the GOODS persistent class library for Java.

You should choose a synchronization strategy to control concurrent access to the objects of your classes. By default, the pessimistic repeatable read scheme is used in GOODS. This scheme provides the highest level of isolation, but causes significant communication overhead and is deadlock prone. You can use a pessimistic or even an optimistic scheme or develop your own metaobject class. It is possible to assign a metaobject to the object instance explicitly (but this binding will remain only until the object is in the client's cache); or associate a default metaobject with the object class (so that it will be used for all instances of the class). If the class definition has static final components of Metaobject type, then the GOODS client library treats the value of this field as a default metaobject for this class and all derived classes (unless they specify their own default metaobjects).

Database servers are accessed by client applications through the Database class. A database is opened by the Database.open(String config_File) method, which takes the name of the database configuration file as its parameter. This file can be accessed via NFS from a central server or can be distributed to client computers through some other mechanism. The first line of this file contains the number of storages in the database. All successive lines specify locations of storage servers. Each line consists of three parts, separated by a colon: storage identifier, host name and port number. The storage identifier can range from 0 to number-of-storages -1.

After the successful database opening, it is necessary to extract the root object. Each GOODS storage has a single root object. It can be retrieved by the Database.getRoot(int sid) or Database.getRoot() method. The last one gets the root of the storage 0. If the storage was not yet initialized, the getRoot() method returns null. In this case it is necessary to create a root object and assign it to the storage by means of the Database.setRoot() method. To prevent concurrent initialization of storage by several clients, the root object is locked by the Database.getRoot() method (only if null is returned) and the lock is released only by the Database.setRoot() method. The method Database.setRoot() can be invoked only once and should not be used for already initialized storage.

Having a reference to the root object, you can access any other object in the database using standard facilities of the Java language. Newly created objects of persistent capable classes will become persistent when you store references to them in some other persistent object. By default the new object will be placed in the same storage as the persistent object, which references it. But it is possible to explicitly assign the storage to the object by means of the Persistent.attachToStorage(Database db, int sid) method. This method can not be applied to an already persistent object in order to reallocate it in another storage. The object will become persistent after commit of transaction.

GOODS interface for Java uses a model of implicit transaction commit: each invocation of a method of a persistent capable class is considered as a nested subtransaction (without the possibility to undo changes). The transaction is automatically committed when all nested subtransactions are terminated (in other words, when there are no more active methods of persistent capable classes). It is possible to explicitly start and finish a nested transaction by means of the BasicMetaobject.beginNestedTransaction() and BasicMetaobject.endNestedTransaction() methods.

GOODS can notify the application about the deterioration of an object instance (if it was modified by some other client). Notification is done through an object of the goodsjpi.CondEvent class. Clients wishing to receive notification about the object modification should invoke the Database.notifyOnModification(Persistent obj, CondEvent event) method and create a separate thread, which will wait until the event object will be signaled using the CondEvent.waitSignal method. To cancel the delivery of notifications, pass a null value as the event parameter.

If the optimistic synchronization strategy is used, it is possible to receive a notification about transaction abortion. This can be done by the Database.notifyOnTransactionAbort(CondEvent event) method.

After finishing work with the database, you should call the Database.close() method to finish the database session. If a client session was aborted by a server (due to server shutdown or server/communication failure), then the Database.disconnected(int sid> method is called, which throws the goodsjpi.SessionDisconnectedError exception. Other errors, detected by the GOODS client interface, are handled by the Database.handleError(String text) or the Database.handleException(IOException x) method. By default, these methods just raise an exception signaling a server communication failure. The application programmer can redefine these methods by creating her/his own class derived from the Database class.

So the template scheme of a Java GOODS application is the following:

import goodsjpi.*;

class MyRoot extends Persistent { 
    ...
}

class Application { 
    static public void main(String args[]) { 
        Database db = new Database();
	if (db.open("app.cfg")) { 
	    MyRoot root = (MyRoot)db.getRoot();
	    if (root == null) { 
	        root = new MyRoot();
		db.setRoot(root);
	    }
	    ... // do something with the database
	    db.close();
	} else { 
	    System.err.println("Failed to open database");
	}
    }
}

After compiling your application, you should run the JAVAMOP preprocessing utility. You should specify a complete list of all Java class files, used in your application, together with goodjpi.jar (and goodslib.jar, if used). You can see examples of running the JAVAMOP tool in GOODS' makefiles. JAVAMOP also has two optional parameters. The option "-package package-name" specifies the name of the package containing the definition of the Metaobject. You should always use "-package goodsjpi" for GOODS applications. The second option is -public and should be used only with JDK 1.1. With this option present, JAVAMOP will change access attributes of classes and fields to PUBLIC. The reflection package in JDK 1.1 doesn't allow to access non-public components: thus loading objects from the database is impossible. You can certainly declare all classes and their fields as public in the Java sources yourself; but it is better to preserve the class encapsulation at source level and use the -public option to solve problems with the JDK 1.1 reflection restrictions. The following command line illustrates the usage of JAVAMOP:

javamop -public -package goodsjpi *.class \goods\goodsjpi.jar \goods\goodslib.jar

resp.
javamop -public -package goodsjpi *.class /goods/goodsjpi.jar /goods/goodslib.jar

1
0: localhost:6100

Then you should start the database server(s) at the specified net node(s). The server is started by the command "goodsrv database-name [storage-identifier]". If the storage identifier is not specified, the storage 0 is used by default. For example, the GOODS server for this template application can be started by the command "goodsrv app". More information about the goodsrv program can be found in "readme.htm" document. Having started the server(s), you can run the application(s).

For some applications client-server architecture is not needed. They are using GOODS as embedded database engine. For such applications GOODS provides goodsjpi.Server class which makes it possible to start/stop GOODS server from the application. You should create this class with specifying database configuration file name to the constructor and then use start/stop methods. You can also send any command to the server process using BufferedReader Server.command(String cmd) method. When you are launching your Java you should not forget to add GOODSRV to PATH environment variable. See Guess.java example.

It is possible to change the definition of classes without loosing instances of these classes already stored in the database. GOODS supports automatic online schema evaluation. You can add new fields to the class, delete some fields, change primitive types of fields (any conversions between primitive types are allowed). Reformatting the object instances according to the new format will be done in lazy mode. When the client loads the object, the descriptor of the application class is compared with the descriptor of the class in the database. If the descriptors are different, the GOODS client library performs automatic conversion of the object to the new format. After being modified , the new object will be stored in the storage in the new format. It is possible for several clients to have different class definitions and work simultaneously with the same database. Unfortunately, GOODS' automatic update mechanism is not able to rename fields, to change the types of references and to change the inheritance graph.

Using Interface types and the PersistenceFactory

As of 2.51, to increase the flexibility of the GOODS implementation you can use Interface types in your objects, such as

  Set foo;

  PersistentHashSet foo;

     import persistence.PersistenceFactory;

     public class Bar
       {
       Set foo;

       public Bar()
         {
         foo=PersistenceFactory.get(Set.class);
         }
       }

persistence.dbname=goodstest
persistence.roottype=test.GoodsTest
persistence.java.util.Set=goodslib.PersistentHashSet
persistence.persistence.PersistenceFactory=goodsjpi.GoodsFactory
goodsjpi.meta=goodsjpi.PessimisticMetaobject Optional GOODS specific parameter

     Set foo=PersistenceFactory.get(Set.class,new Class[] {Integer.TYPE, Float.TYPE}, new Object[] { new Integer(100), new Float(0.6f) });

class MyThread extends PThread { 
    public void run() { 
	Database db = new Database();
	db.attach();
	try { 
	    if (db.open("myserver.cfg")) {
	        ...
                db.close();
	    }
	} finally { 
	    db.detach();
	}
    }
}

class MyServer { 
    static public void main(String args[]) throws Exception { 
	int nThreads = nThreadsByDefault;
	Thread[] threads = new Thread[nThreads];
	for (i = 0; i < nThreads; i++) { 
	    threads[i] = new MyThread();
	    threads[i].start();
	}
	for (i = 0; i < nThreads; i++) { 
	    threads[i].join();
	}
    }
}

class MyServelet extends HttpServlet { 
    public void init() throws ServletException {
        // no database initialization is done at this stage
    }
   
    public void service(HttpServletRequest req, HttpServletResponse resp) 
      throws ServletException, IOException 
    {
         Database db = Database.getDatabase("mydb.cfg");
         try { 
	      // work with database db
         } finally {  
 	      db.close();
         }
    } 

    public void destroy() {
         Database.closeAll(); // close all opened databases
    }
}

      class string { 
         protected:
	   int  size;
	   char body[1]; // component with varying size 
      };

     class string {
         protected int  size;
         protected byte body[];
     } 

     class substring extends string { 
         protected int  offs;
     }