Core Modules

Home Project Closed Site Index News Recent Old Download Documents User Docs Development Introduction to ANet High-Level Design Run-Time Wrapper Client Connection Modules Data Transformation Modules Client SDK Cluster Group Modules Cluster Filter Modules Core Modules Handshaking Protocol Modules Packet Protocol Modules Connection Protocol Modules Bandwidth Manager Module Document Type Definition (DTD) Low-Level Design Protocol Specifications Task List Development Roadmap Other Docs Development Mailing Lists Contacting Us

PROJECT CLOSED

by Benad

The Core Modules are doing the actual data distribution within a network.

Table of Contents

• Design
  • Why Core Modules?
  • Input/Output of the Core Modules
  • Core Modules
    • Query Module
    • Static Data Module
    • TWDT Module
  • Managing Connections
• Implementation Notes
• DTD

Design

Why Core Modules?

We need to distribute data somewhere, so we need the Core Modules, right? Actually, if data distribution is so important, how come the most important part of the ANet Protocol resides in external modules?

There is a simple answer: the rules of distribution themselves can be changed, and new distribution methods could be added. For example, let's say that we just found a way to optimize the distribution of Static Data. Then, the only thing we need to install is a newer version of the Static Data Module. No painful version upgrade is needed.

Input/Output of the Core Modules

The "main" function of each module is called with both a list of packets to distribute and a list of commands that were called by the clients. The list of packets is destroyed between each function call. Commands will also be deleted, but only if you mark them as "was used". All "used" commands will be deleted, while the other ones will be sent in a similar manner to the other Core Modules. Unused commands will be finally deleted if they were unused by all the Core Modules.(1)

The input packets are AIPs. There is one buffer for all the cluster, thus it is up to the Core Module to figure out to which connections the packets should be re-transmitted.

As output, the Core Module will have to produce one buffer per connection ID. The output packets must be ANet External Packets (AEP), which are totally platform independant. The deamon will provide some functions to transform an AIP to an AEP, though the modules might have to set some values by themselves(2).

Core Modules

Here is a high-level description of the Core Modules that will be implemented for the first versions of ANet.(3)

Query Module

This module will implement queries[1] for ANet. A query packet is sent to a connection only if the same packet was not sent recently and if the "Time-to-Live" value of the packet is not 0. Also, the packet must not be re-sent to the connection that originally sent the packet.

To do that, the Query Module will keep a list of the packets recently sent, in the form of a simple checksum(4) and the origin of each packet.

Static Data Module

This module will implement Static Data[2] in ANet. It will mostly use the hard disk to store the data, though it might need a larger cache than the Query Module to keep its indexes and recently used static packets.

TWDT Module

This module will implement Anonymous Two-Way Data Transfers (TWDT[3]) in ANet. Its input come only from the client commands[4] made for the TWDT module.

This module will need to use the Query Module through Inter-Module Communication[5], even though the caching of the queries it needs to produce is handled by the TWDT Module itself.

Managing Connections

As input, the modules will always have a list of the connections, and their state. That list is kept in the parsed configuration file[5]. Here are the possible states.

Open

The connection is open, thus it will accept input data and produce output data.

On Hold

The connection cannot do any input or output, but will soon become either Open again or be Closed.

Closed

The connection is forever closed. The connection will soon be removed from the connection list.

The Core Modules cannot, by themselves, change the state of a connection. Only Cluster Filter Modules[6] and by Handshaking Protocol Modules.

Implementation Notes

• Be careful implementing Core Modules. If they don't work, nothing will happen. The entire ANet deamon depends on the implementation of the Core Modules (hence the name "Core").
• The Core Modules are not limited to the three basic Core Modules that will be first implemented (Query, Static and TWDT). If you want the distribution to be based on other rules, then you should implement new Core Modules.

DTD

See the complete DTD for more information.

<!-- Core Modules. Simply contains a list of CoreModule elements. -->
<!ELEMENT CoreModules (CoreModule)+>
<!ATTLIST CoreModules %security;>

<!-- Core Module -->
<!ELEMENT CoreModule EMPTY>
<!ATTLIST CoreModule %moduleName; %args; %security;>

Notes

(1) The Run-Time wrapper[5] will allow you to create as many new commands as you want. Actually, commands that are unrecognized by the wrapper will be sent to the Core Modules.

(2) For example, the "Time-to-Live" exists only in the AEPs (for query AEPs), so it is up to the Query Module to fill the value. Note that the format or the AEPs will be very similar to the format of the AIP, though without memory alignment and with strict rules for byte ordering.

(3) The modules were already described in the development introduction. They don't really have a high-level design, and the low-level design will be covered during development. That's because it's too simple (Query Module) or too complex (Static Data and TWDT Module) to be worth the time investment. Don't worry, the modules will be thoroughly documented once a stable implementation will be done.

(4) By simple, I mean easy to compute. For example, MD5[7] checksums would take too much CPU, and isn't very good for small data anyways.

References

About the references...

Last update for this document: August 24, 2001, at 1:22:47 PST