Requests from client to server

Replies from server to client


Common issues

GOODS uses the same format of message headers for all requests from client to server and server responses and notifications. Message header is 32-byte long, size of first two fields is one bye, size of third field is two bytes, and size of all other three fields is four bytes. Last field is not currently used by GOODS protocol of client-server interaction, and so it can be used for protocol extension.

All data is passed in network format (big endian) as unaligned stream of bytes (as in Java). Object reference occupies 6 bytes, first 4 of them specify object identifier, which is unique for the storage, and last two bytes specify storage identifier. Class identifier is two bytes long and is also unique for the storage.

Objects are passed as unaligned sequence of fields. All reference fields should be placed before all non-reference fields (such order increase performance of many server algorithms, such as garbage collector, ...). Only object fields described in the object class descriptor should be passed to the server.

Most of servers messages are synchronous and are sent as replies to client requests. There are only two notification messages: cmd_invalidate and cmd_bye, which can be sent asynchronously by the server, so client should be ready to receive them in any moment (but this notification message can't interrupt another message, so all message data is arrived sequentially). Client should not sent other messages until it receives reply to previous message (if it requires reply). So multithreaded client application should use some synchronization discipline to provide consistent access to the storage server by different threads.

Requests from client to server

cmd_load: load object from server

byte 0byte 1byte 2byte 3
0-load flags
object identifier
-
-

Request to load object or cluster of objects from the server. Server will reply with cmd_object message containing instance of requested object (and my be some other object from the same object cluster).

load objects flags
flagmeaning
1Client request another copy of object. By setting of this flag client can load most recent version of object while still keeping old version. If this flag is not set, server consider that client replace old version of the object with new one.
2Allow sending cluster of objects together with requested object. Strategy of including objects in the cluster depends on server. This flag is normally set, unless object is reloaded.

cmd_forget: client remove reference to loaded object

byte 0byte 1byte 2byte 3
2--
object identifier
number of following notification requests
-

Client informs server that there s no more reference to specified object from client application. Server will remove association between this object instance and client process. Client can transfer more than one notification requests in one send operation. If field 5 of the request is not zero, server will read specified extra number of requests. This request should be only of cmd_forget or cmd_trow type. Value of filed 5 of all this extra requests is ignored.

cmd_throw: client throw away instance of object from cache

byte 0byte 1byte 2byte 3
3--
object identifier
number of following notification requests
-

Client informs server that object was thrown away from client cache as a result of LRU cache management discipline or because it was modified by some other process. Server can remove old version of object from the storage only if all clients throw deteriorated instance of this object from the cache. Client can transfer more than one notification requests in one send operation. If field 5 of the request is not zero, server will read specified extra number of requests. This request should be only of cmd_forget or cmd_trow type. Value of filed 5 of all this extra requests is ignored.

cmd_getclass: get information about class from server

byte 0byte 1byte 2byte 3
5-class identifier
-
-
-

Load class descriptor from the storage by specified class identifier. Server will reply with cmd_classdesc message, containing class descriptor with specified identifier.

cmd_putclass: register new class at server

byte 0byte 1byte 2byte 3
7--
size of class descriptor
-
-

Request to look for descriptor of the class in the storage. If there is no class with such descriptor in the storage, then new class identifier is assigned to this class and it's descriptor is stored in the storage. Server will reply with cmd_classid message, containing identifier assigned to this class. Body of class descriptor is sent immediately after this request header. Format of class descriptor header is the following:

Class descriptor header is followed by specified number of field descriptors. Format of class descriptor is described below:
byte 0byte 1byte 2byte 3
Size of fixed part of the class
Size of varying part of the class
Number of references in fixed part of the class
Number of references in varying part of the class
Number of all fields in the class including subclasses
Size of class name dictionary - total size of all class fields names
byte 0byte 1byte 2byte 3
type of the fieldoffset to field name
Size of the field
Number of components
Index of the next field in the class

Field "number of components" should contain:

Field types
valuemeaning
0structure
1object reference
2signed integer
3unsigned integer
4real number

cmd_modclass: modify existed class at server

byte 0byte 1byte 2byte 3
9-class identifier
size of class descriptor
-
-

Modify descriptor of existing class in the storage. This request can be used by administration utilities to rename class fields. It should not be used by normal application, because GOODS automatically handles scheme evaluations using fields names to establish mapping between different version of classes. This request header is followed by class descriptor.

cmd_lock: request to server from client to lock object

byte 0byte 1byte 2byte 3
10lock typelock attributes
object identifier
-
-

Lock object in specified mode. Possible values of lock type is 1 for shared lock (prevent object from been modified or exclusively locked by other processes) or 2 for exclusive lock (prevent object from been modified or locked by other processes). The only supported lock attribute is "no wait" attribute with value 1. If "no wait" attribute is set and granting requested lock is not possible, negative reply will be returned immediately, otherwise request will be blocked until lock will be set. Server will return client about the result of request by means of cmd_lockresult message.

cmd_unlock: request to server from client to unlock object

byte 0byte 1byte 2byte 3
12lock typeload flags
object identifier
-
-

Unlock previously locked object. In this request lock type specify mode to which lock should be downgrade. To release lock completely pass 0 as lock type.

cmd_transaction: client sends transaction to coordinator

byte 0byte 1byte 2byte 3
13-number of servers
transaction size
-
-

Send local transaction to the server or part of global transaction to coordinator. Protocol of local and global transaction fixing protocol are described in the section Transaction protocol, Transaction is passed to the server as a sequence of object headers. If object was modified by transaction then object header is followed by modified object instance. Modified objects should be placed before non-modified objects. It is necessary to pass headers of non-modified objects to the server to validate these object state and unlock locked objects. Format of object header is specified in the following table:

byte 0byte 1byte 2byte 3
transaction flags
object identifier
object class identifier object storage identifier
object size

Transaction flags in object header specify action performed by server for this object while committing transaction. This field is bit combination of the following flags:

ValueMeaning
1update the object (object header should be followed by modified object instance)
2validate the object (check if client has the most recent version of the object)
4unlock the object (remove all locks granted by the client)
8downgrade exclusive lock of the object to shared lock

cmd_subtransact: client sends local part of transaction to server

byte 0byte 1byte 2byte 3
14-coordinator of transaction
transaction size
transaction identifier
-

Send part of global transaction to the server, which is not coordinator of this transaction. Details of global transaction fixing protocol are described in the section Transaction protocol, Format of transaction body, passed after this request header, is the same as in cmd_transaction request.

cmd_login: client login at server

byte 0byte 1byte 2byte 3
17--
client name length
-
-

Open client session at the server. Login name of the client is passed immediately after this request header. Server will perform authorization and reply with cmd_ok or cmd_refused messages.

cmd_logout: client finish the session

byte 0byte 1byte 2byte 3
18--
-
-
-

Request from the client to terminate client session at the server. Server will reply with cmd_bye response message.

cmd_alloc: client request server to allocate object

byte 0byte 1byte 2byte 3
23-class identifier
size of object
-
-

Allocate object of specified class and size at the server. Server will return identifier of created object by means of cmd_location message.

cmd_free: client request to free object

byte 0byte 1byte 2byte 3
25--
object identifier
-
-

Request to explicitly deallocate object. As far as GOODS is using garbage collector to free unreferenced objects, this request should be used with case only when client is sure that there are no references to deleted object in the storage (and in other clients). In current GOODS implementation this request can be send during abort of transaction for objects created by this transaction.

Replies from server to client

cmd_object: server send object to client

byte 0byte 1byte 2byte 3
1--
size of object(s)
-
-

Server send this message as a reply to cmd_load request. This message header is followed by one or more objects (if sending cluster of objects was allowed in the request). Field 4 specifies total size of all objects in this message. Each object is represented by objects header followed by the object body. Object header has the same format as described in cmd_transaction request:

byte 0byte 1byte 2byte 3
-
object identifier
object class identifier object storage identifier
object size

cmd_invalidate: notification about object modification

byte 0byte 1byte 2byte 3
4--
object identifier
number of extra notification messages
-

This message is send asynchronously to the clients by the server. When some object is changed by some transaction, server send notification about object modification to all clients having instances of this object (except client initiating this transaction). Several notifications can be included in the same message. In this case field 5 specifies number of extra notifications, following the message header. These notifications have the same format as message header, except that filed 5 of should be ignored.

cmd_classdesc: server send class descriptor requested by cpid

byte 0byte 1byte 2byte 3
6--
size of class descriptor
-
-

This message is a reply to cmd_getclass client request. Class descriptor is passed in the same format as described in cmd_put_class reply.

cmd_classid: class identifier returned to client by server

byte 0byte 1byte 2byte 3
8-class identifier
-
-
-

This message is a reply to cmd_putclass client request. Server returns to the client identifier assigned to the class, passed in cmd_putclass request.

cmd_lockresult: result of applying lock request at server

byte 0byte 1byte 2byte 3
11lock result-
-
-
-

This message is response to cmd_lock request. Field 1 is set to 1 (TRUE) if lock is successfully granted, otherwise it is set to 0 (FALSE).

cmd_transresult: coordinator returns transaction status to client

byte 0byte 1byte 2byte 3
15status-
-
global transaction identifier
-

This message is send by coordinator of transaction or by server of local transaction to the client with status of transaction fixing. For global transaction, coordinator first assign global identifier to the transaction and send cmd_transresult message with this identifier to the client. Client then sends local parts of global transaction to all other servers, involved in transaction by means of cmd_subtransact message. Then servers perform global commit of transaction using two-phase protocol and than coordinator inform client about status of transaction fixing. If transaction is successfully committed, status field of cmd_transresult message is set to 1, if transaction is aborted - it is set to 0. In case of local transaction, only one cmd_transresult message is sent to the client, containing status of fixing transaction at the server.

cmd_bye: notification about server shutdown

byte 0byte 1byte 2byte 3
20--
-
-
-

This message can be send to the client either asynchronously, when server shutdown is initiated by database administrator, or synchronously - as the reply to cmd_logout request.

cmd_ok: successful authorization

byte 0byte 1byte 2byte 3
21--
-
-
-

Positive reply to cmd_login client request. It is send when client successfully pass authorization check. Only in this case database session is really created for the client.

cmd_refused: authorization procedure is failed at server

byte 0byte 1byte 2byte 3
22--
-
-
-

Negative reply to cmd_login client request. It is send when authorization procedure is failed for this client. No database session is opened for the client in this case.

cmd_location: address of allocated object

byte 0byte 1byte 2byte 3
24--
object identifier
-
-

This message is reply to cmd_alloc request. Identifier of object created in this storage is returned back to the client.

Transaction protocol

All object modified within transaction should be stored in client cache until the end of transaction. When transaction commit is initiated, all modified objects and may be headers of all objects accessed within transaction (it depends on transaction isolation level) should be send to servers of storages where these object are located. If all objects belong to the same storage then transaction considered to be local and requires no special protocol for fixation. Otherwise (if objects from several storages are involved in the transaction), transaction considered to be global and requires two-phase fixation protocol.

In case of global transaction, first of all coordinator of this transaction should be chosen. Current agreement in GOODS is to use server with the smallest identifier among all servers involved in transaction as coordinator of this transaction. Such agreement is significant for recovery procedure which should restore servers in special order. Then client sends part of global transaction to coordinator (objects located at coordinator storage should be included in this part). Format of passing transaction is specified in cmd_transaction. Here I only want to notice, that modified object should be placed before headers of non-modified objects (such order allows server not to reorder records in transaction buffer). Coordinator will reply with cmd_transresult message, containing global identifier of transaction assigned to the transaction by coordinator. Also status field should be checked in this message. By some reasons transaction can be aborted by the coordinator at this stage. If status is TRUE, client should send local parts of transaction to all other servers, including in cmd_subtransact message global identifier of transaction assigned by coordinator. After it, servers will perform two-phase transaction fixing protocol and result will be returned by coordinator to the client by cmd_transresult message.

In case of local transaction, only one cmd_transaction request should be sent to the server. Server will reply with cmd_transresult message, informing client whether transaction is successfully committed or aborted.

Two ensure transaction serializability, locks set within transaction, should be kept until the end of transaction. It is possible to automatically unlock objects by passing special flag in header of object in transaction body. If transaction is aborted, locks are not released and client application should unlock objects itself, by cmd_unlock requests.