tok_trans API

Warning! This information is currently incomplete. Constructors and destructors needed documenting, for example.

tok_trans modules convert streams of tokens to streams of tokens. Readers are special cases that read no tokens and generate tokens from their input, usually a file.

Mandatory simple methods

All modules must provide a few simple methods. Some of these methods can usually be accomplised in a single statement but sometimes require more complex coding.

• const char *modname(void) returns the name of the module. The usual implementation just returns a constant string. A few modules are more complex: tokwall and tap return an a string allocated at construiction time (due to the need to attach a number), and dlopen checks whether object file provides the method or not. Returning NULL is not acceptable
• int reset() resets all the state of the module bar tunable options, which must remain unchanged. This method must be callable at any time with any internal state. The method should de-allocate any memory that has been allocated during processing. The method returns 1 if the reset succeeded and 0 otherwise.

The feed_tokens() method

int feed_tokens(void) is the main method. The methods should return 0 at the end of a token stream and 1 at all other times. See the reading and outputing sections below for detial of these operations.

Any token that the method obtains from the previous stage should be either freed or passed on. Forgetting about tokens and creating vast memory leaks is not acceptable. If the previous stage is NULL, which modules may assume is not the case unless there is a good reason yo assume otherwise, must not use the feed(), read(), or any related method.

Modules must allocate new tokens using the new() method. Using static data is not possible, due to the rule that methods must either free or pass on tokens. All tokens that reach the output code are freed after use.

The feed_token() method must may queue any non-negative number of tokens for each call. The method must never return 1 and queue no tokens indefinately.

Outputing tokens

All tok_trans modules output tokens by putting tokens on their output fifo. Once tokens have been inserted omto the output fifo they have been output irrecovably; once a token has been queued on the poutput file the next stage has an inalienable right to remove it from the queue.

Use a seperate fifo to hold output that might need something prepended, modifcation or destruction instead of mkaing assumptions about the next stage. Modules must not make assumptions about the next or previous stages. list.cc does this when it sees items that might or might not constitute a list.

See the seperate information on fifos for detials of avialable methods of adding items to fifos.

Reading tokens

const tok * items are read using the feed() method. feed() returns NULL at the end of a file. Remember that it must be possible to use the reset() method when the module is in this state.

Modules may push tokens back using the push_back method. The fifo<tok> or const tok * pushed back is prepended to the input fifo and thus will appear before any new input. In particular this means that reading two tokens and pushing them back in the same order they were read will return them will feed() them back in reverse order. This method depends on the previous stage not being NULL, because it manipluates the previous stage's output fifo.