ToC | DocOverview | CGDoc | RelNotes | FAQ | Index | PermutedIndex
| Allegro CL version 9.0 Unrevised from 8.2 to 9.0. 8.2 version

# Gray Streams in Allegro CL

This document contains the following sections:

1.0 Introduction to Gray streams in Allegro CL
2.0 Documenting object-oriented protocols
3.0 Stream classes
4.0 Generic functions for character input
5.0 Generic functions for character output
6.0 Generic functions for binary streams
7.0 Functions for efficient input and output of sequences
8.0 Creating streams
9.0 Miscellaneous stream functions

Since release 6.0, Allegro CL has a new streams model using simple-streams, which have no element-type, but which act as if they have element type `(unsigned-byte 8)`. The new implementation is described in streams.htm. This document describes the Gray Streams implementation in Allegro CL, which was the stream implementation in releases prior to 6.0. The Gray Streams implementation is preserved for backward compatibility.

Gray stream code is not typically included in the Allegro CL image. Load it with `(require :streamc)`.

In general, a Gray stream is created whenever a file is opened (with open) with an element-type specified. If open is called without an element-type specified, a simple-stream is created. If you have code which does not specify an element-type, and you want a Gray stream, specify `:element-type 'character`. Do note that you cannot assume that system-created streams will be Gray streams in version 6.1 or later.

String output streams are always simple-streams even though both make-string-output-stream and with-output-to-string accept an element-type keyword argument. Both operators create simple-streams regardless of whether a value is specified for the element-type keyword argument or not.

The transition from Gray streams to simple-streams should be easy and transparent unless you have done extensive stream customization.

The rest of this document is more or less unchanged from release 5.0.1. It describes the Gray stream implementation.

Symbols naming gray stream functionality are generally in the `:excl` package.

## 1.0 Introduction to Gray streams in Allegro CL

Streams in Common Lisp have always been first-class objects and the stream type a first-class type. The inability of user code to customize or extend stream behavior has been an unfortunate limitation especially because the capabilities of the Common Lisp reader and printer can only be obtained through the interface of a stream. Suppose, for example, one wants to do normal Lisp printing while performing some simple output character translations. The only portable way to do this -- that is, without knowing about each implementation's internal functions -- would be to rewrite the whole printer from scratch for the single purpose of interposing a translation function around every call to write-char and write-string. This is clearly unacceptable. It should be possible to customize a stream to add such simple behavior without reimplementing large portions of Common Lisp.

When Common Lisp was first defined the Common Lisp Object System (CLOS) was not yet conceived. Some portable stream construction utilities were originally provided by the several "indirect" stream types - synonym-stream, concatenated-stream, etc. - but these provide only very limited kinds of customization. The subsequent ANSI standard for Common Lisp includes CLOS, so that is now the obvious mechanism to support user-customizable stream types.

Fairly late in the ANSI standardization process there was some consideration of closifying streams. The most complete proposal was by David N. Gray, then of Texas Instruments. Gray's proposal was only a draft and it acknowledged a number of problems and omissions. It was intended only as a starting point towards a complete specification. However, Gray and the standardization committee soon decided it best not to act because the change to the language would be large (causing a lot of work for implementors) and also because there was little actual experience with implementations of CLOS streams. Despite the feeling that it was not yet the time to adopt closified streams into the language standard, most members of the committee feel that redefining streams in terms of CLOS would be intrinsically worthwhile, and experimentation with extensions should be encouraged.

The implementation of streams in Allegro CL from release 4.0 (on Unix) through release 5.0.1 (on Unix and Windows) is based largely on the X3J13 issue writeup entitled "STREAM-DEFINITION-BY-USER, Version 1, 22-Mar-89 by David N. Gray." Some of the text in this chapter is taken from that proposal, but there are numerous additions, modifications, clarifications, and comments specific to the Allegro implementation.

An important feature of the Gray proposal is that it is upward compatible with both the current language standard and earlier implementations of Allegro CL. Programmers need not know anything about closified streams unless they actually use its features.

The symbols naming classes and generic functions for the CLOS Gray stream interface are exported from the `excl` package.

The existing Common Lisp I/O functions cannot be made generic because in nearly every case the stream argument is optional and therefore cannot be specialized. It is therefore necessary to define new generic functions which are called internally by the standard Common Lisp functions. In order to make the meaning as obvious as possible, the names of the generic functions have been formed by prefixing "stream-" to the corresponding non-generic function (e.g. terpri and stream-terpri). Note that for all of these generic functions, the stream argument must be a stream object, not `t` or `nil`.

Having the generic input functions consistently return `:eof` at end-of-file, with the higher-level functions handling the eof-error-p and eof-value arguments, simplifies the generic function interface and makes it more efficient by not needing to pass through those arguments. Note that the functions that use this convention can only return a character or integer as a stream element, so there is no possibility of ambiguity.

## 2.0 Documenting object-oriented protocols

There is much uncertainty in the industry about how to document an object-oriented protocol. There are many ways to do it wrong, and it is far from clear how to do it right. Of course, problems with documentation can often be ascribed to poor design of the underlying system itself or else design based on assumptions that are not made explicit.

For example, the default method for stream-write-string is defined to do repeated calls to stream-write-char. The character-translation problem mentioned at the beginning of Section 1.0 Introduction to Gray streams in Allegro CL above could be implemented minimally by defining an `:around` method for stream-write-char, and everything ought to work. But what if some stream specialization -- say, the `excl::file-gray-stream` classes provided by the implementation -- optimizes out the repeated calls to stream-write-char in the interest of efficiency? The Gray proposal is silent whether this would conform, but it is clear that independently-written mixin classes will need to know each other's assumptions about which publicly-specializable generic functions do or do not call each other's publicly-specializable generic function. There is also controversy whether there should be a default method for a particular generic function or whether a method definition should be required on specialized stream classes. The difference is important if mixin methods normally do a call-next-method, because the question is left open whether the default method should be, should not be, or may optionally be shadowed.

In attempting to answer these questions over the years, we have decided that this is not the best approach, and have opted to redesign streams per documentation in streams.htm.

This said, there are several kinds of information this document needs to specify:

1. The public superclasses (base and mixin) that may be used to define customized stream classes.
2. The generic functions that are defined on these classes, when and how they are called, and which ones call others.
3. The methods that are defined on these generic functions by the implementation, and what functionality is required if the user chooses to override any builtin methods.
4. Which builtin methods and built-in nongeneric functions call other public generic functions.
5. Which methods are required of any specialization of a class that are not provided by default methods on base classes.
6. Which public superclasses (if any) are required to be mixed into a user-customized class.

It is a legitimate criticism that this chapter does not exhaustively cover all these details. The legitimate excuse is that cogent, defensible design decisions have not yet been made for all of them. In particular, there is something of a trade-off between (5) and (6) depending on whether one believes subclassing should be defined in terms of class inheritance or in terms of generic function behavior protocol. The new implementation described in streams.htm addresses these issues in a different way.

## 3.0 Stream classes

Two kinds of classes are mentioned here. Some are "mixin" classes intended to be used as super classes of user-defined stream classes. They are not intended to be directly instantiated; they primarily provide places to hang default methods. Others are classes actually instantiated by Allegro CL, for example, to service a call to open.

Those classes that are sufficiently complete to be meaningfully instantiated are labeled as Instantiable Class in their description pages while mixin classes are labeled simply as Class. You can, of course, further subclass both kinds of classes. Although most stream classes have their own description page, the pages do not contain more information than is present in this document and so we have not provided links.

Table of mixin (non-instantiable) stream classes
Class | Notes
`fundamental-stream` | This class is a subclass of `stream` and of standard-object. streamp will return true for an instance of any class that includes this.
`fundamental-input-stream` | A subclass of `fundamental-stream`. Its inclusion causes input-stream-p to return true. Note: any user-defined stream class that will do input must include this class. Bidirectional streams may be formed by including subclasses of both `fundamental-output-stream` and `fundamental-input-stream`.
`fundamental-output-stream` | A subclass of `fundamental-stream`. Its inclusion causes output-stream-p to return true. Note: any user-defined stream class that will do output must include this class. Bidirectional streams may be formed by including subclasses of both `fundamental-output-stream` and `fundamental-input-stream`.
`fundamental-character-stream` | A subclass of `fundamental-stream`. It provides a method for stream-element-type which returns `character`.
`fundamental-binary-stream` | A subclass of `fundamental-stream`. The Allegro CL implementation requires the `:element-type` keyword be provided to make-instance for streams of this class.
`fundamental-character-input-stream` | Includes `fundamental-input-stream` and `fundamental-character-stream`. Any user-defined stream class that will be used as an argument to read and friends must include this class.
`fundamental-character-output-stream` | Includes `fundamental-input-stream` and `fundamental-character-stream`. Any user-defined stream class that will be used as an argument to print, format, etc. must include this class.
`fundamental-binary-input-stream` | Includes `fundamental-input-stream` and `fundamental-binary-stream`.
`fundamental-binary-output-stream` | Includes `fundamental-output-stream` and `fundamental-binary-stream`.
`file-gray-stream` | See just below.
`excl::string-gray-stream` | See just below.

ANSI Common Lisp mandates these seven subtypes of type `stream`: `file-stream`, `string-stream`, `echo-stream`, `concatenated-stream`, `two-way-stream`, `broadcast-stream`, and `synonym-stream`. The first two are mixin classes and the rest are instantiable classes. `file-stream` has two subclasses used in Allegro CL: `excl::file-gray-stream` and `excl::file-simple-stream`. `string-stream` also has two subclasses: `excl::string-gray-stream` and `excl::string-simple-stream`.

Since in Allegro CL's implementation stream is a CLOS class (more precisely, a subclass of `clos:standard-object`) then the subtypes of `stream` must also be CLOS classes.

The stream types other than `file-stream` and `string-stream` are sometimes called indirect streams. The creator functions for these stream types (make-synonym-stream, make-echo-stream, make-broadcast-stream, make-concatenated-stream, and make-two-way-stream) are unmodified from the standard Common Lisp definitions. These stream types represent an early (and awkward) attempt in Common Lisp to obtain part of an extensible stream facility. These stream types may not be defined in an image but the module defining them, streama, will be loaded if it is require'd or if `(find-class 'x)` is evaluated or make-x is called (where x is one of the classes). streama is also loaded automatically if any of the Common Lisp slot-readers for one of these streams is called. (streama stands for stream-ansi. The other module is streamc, which originally stood for stream-clos, but that interpretation is no longer appropriate since yet another stream module, streamd, is also CLOS-based (so `c' and `d' are single letter file indentifiers only). streamd implements simple-streams. streamc is only loaded into a Lisp when gray streams are used. Gray streams include the indirect streams -- synonym-stream, concatenated-stream, etc.)

The generic function approach does not integrate very well with indirect streams because the indirect stream classes cannot anticipate the full set of generic functions that user code may want to define over all streams in order to indirect them to the contained streams. It is arguable that no purely automatic mechanism can handle this, not even by defining a general method for `no-applicable-method`. For example, consider the problem a two-way-stream has choosing whether to pass to its input stream, output stream, or both a call to a generic function about which it knows nothing. For this reason there has been no attempt to extend or make customizable the five indirect streams.

The five indirect stream classes are instantiable, but at present cannot successfully be instantiated other than with their standard constructor functions (e.g. make-synonym-stream).

The `excl::file-gray-stream` class is customizable since it embodies the interface to the file system. `excl::file-gray-stream` is a mixin not intended to be instantiated directly. Its subclasses are normally instantiated by the open function (see below) although it is also possible to create them directly with make-instance.

Table of instantiable stream classes
Class | Notes
`cl:echo-stream` | See discussion above. Created with cl:make-echo-stream.
`cl:concatenated-stream` | See discussion above. Created with cl:make-concatenated-stream.
`cl:two-way-stream` | See discussion above. Created with cl:make-two-way-stream.
`cl:broadcast-stream` | See discussion above. Created with cl:make-broadcast-stream.
`cl:synonym-stream`. | See discussion above. Created with cl:make-synonym-stream.
`input-terminal-stream` | All six classes implement streams intended to support connection to sockets and input-output devices that connect to a "stream" of data rather than a fixed file stored in a file system. A normal call to open without the `:class` argument extension always creates a file stream. The initial value of `*terminal-io*` at startup is a `bidirectional-terminal-stream` stream. One essential difference between these two groups is evident for the bidirectional versions. The input and output sides of a bidirectional file stream access the same data storage (e.g. on a disk) and so need to cooperate closely about buffering when input and output operations are interleaved. They also share a single file-position pointer. The input and output sides of a socket stream are completely separate channels, and socket streams don't support file position at all. These classes all require an `:fn-in` and/or `:fn-out` initialization argument which should be a small integer that is a Unix file descriptor. (On Windows there is an internal hidden translation between the small integers and actual Windows "handles".) An error will be signaled if the required initializer is omitted. The input and output file descriptor of a bidirectional stream may be and typically are the same.
`output-terminal-stream`
`bidirectional-terminal-stream`
`input-binary-socket-stream`
`output-binary-socket-stream`
`bidirectional-binary-socket-stream`

## 4.0 Generic functions for character input

A character input stream class can be defined by including `fundamental-character-input-stream` and defining methods for the generic functions below. The builtin instantiable classes in Allegro CL already have appropriate methods, of course.

Generic function | Arguments | Notes
stream-read-char | stream | This reads one character from stream.
stream-unread-char | stream character | Undoes the last call to stream-read-char, as in unread-char. Returns `nil`.
stream-read-char-no-hang | stream | This is used to implement read-char-no-hang. It returns either a character, or `nil` if no input is currently available, or `:eof` if end-of-file is reached.
stream-peek-char | stream | Used to implement peek-char; this corresponds to the case where the peek-type argument is `nil`. It returns either a character or `:eof`.
stream-listen | stream | Used by listen. Returns true or false as there is or is not a character available to be read.
stream-read-line | stream | Used by read-line. A string is returned as the first value. The second value is true if the string was terminated by end-of-file instead of the end of a line.
stream-clear-input | stream | Implements clear-input for stream, returning `nil`.

## 5.0 Generic functions for character output

A character output stream can be created by defining a class that includes `fundamental-character-output-stream` and defining methods for the generic functions below.

Generic function | Arguments | Notes
stream-write-char | stream character | Writes character to stream and returns character.
stream-line-column | stream | This function returns the column number where the next character will be written, or `nil` if that is not meaningful for stream. The first column on a line is numbered 0.
stream-start-line-p | stream | This is a predicate which returns `t` if stream is positioned at the beginning of a line, else returns `nil`. It is permissible to always return `nil`. This is called by stream-fresh-line.
stream-write-string | stream string &optional start end | Implements write-string. It writes string to stream, optionally delimited by start and end, which default to 0 and `nil`. The string argument is returned.
stream-terpri | stream | Writes an end of line, as for terpri. Returns `nil`.
stream-fresh-line | stream | Implements fresh-line.
stream-finish-output | stream | Implements finish-output
stream-force-output | stream | Implements force-output.
stream-advance-to-column | stream column | Writes enough blank space so that the next character will be written at the specified column. Returns true if the operation is successful, or `nil` if it is not supported for stream.
stream-clear-input | stream | Implements clear-input.

## 6.0 Generic functions for binary streams

A binary stream class can be defined by including either or both of `fundamental-binary-input-stream` and `fundamental-binary-output-stream`, and methods for one or both of the following generic functions. If you create a binary stream other than by using open you must also include a keyword initialization argument for `:element-type`.

Generic function | Arguments | Notes
stream-read-byte | stream | Used by read-byte; returns either an integer, or the symbol :eof if stream is at end-of-file.
stream-write-byte | stream integer | Used by write-byte; writes the integer to stream and returns the integer as the result.

## 7.0 Functions for efficient input and output of sequences

In the public comments on the first Draft Proposed ANSI Standard several reviewers noted that except for the write-string function, the language provided no means to request efficient input-output on large blocks of data. The ANSI standard consequently added two new functions to the language read-sequence and write-sequence.

Allegro CL also defines generic function versions of these functions. The nongeneric versions are implemented by calling these generic functions. Methods are defined for these functions that handle all legal calls, but not all legal calls will have highly efficient execution. The existing methods will handle vector sequence arguments efficiently, and will also handle start and end arguments efficiently. The Allegro CL implementation will also accept and efficiently transfer data to and from higher-dimension arrays in the usual row-major order, although this is an extension to the language and not defined by ANSI Common Lisp.

Generic function | Arguments | Notes
stream-read-sequence | stream sequence &optional start end | Destructively modifies sequence, storing in it elements read from stream.
stream-write-sequence | stream sequence &optional start end | Writes elements of sequence to stream.

The definition of these functions does not permit the useful ability to operate on sequence elements different from the stream element type. Such capability would clearly be desirable. (It would, for example, permit floats to be transferred efficiently, and allow an application needing to store several different successive sequences of different types to write them to a single stream and later reread the data.) However, the behavior of any such operation would depend on bit representations of objects in an implementation- and machine-dependent way, and would be incompatible with the strict definition of read-sequence and write-sequence, which require automatic conversion of data to the correct element type of the stream or sequence. Any such capability would therefore need to be implemented by a different pair of functions.

## 8.0 Creating streams

Streams of arbitrary class may be constructed by an explicit call to make-instance, but the Gray proposal did not address how to customize the stream created by open. We define a simple interface here. The Gray proposal also omits mention of constructor functions such as make-string-input-stream, make-string-output-stream, and their associated macros such as with-output-to-string. However, there is nothing these various operators do that can't be performed explicitly by user code including a call to make-instance. Unfortunately, Allegro CL's current `string-gray-stream` subclasses do not (reliably) support specialization or even independent instantiation by make-instance. This is a bug in that some required initialization is performed by the make-string-...-stream function.

The open function has been extended to take a class keyword argument. open passes this argument to make-instance when it creates the stream, and as with make-instance, the argument may be a stream class object or a symbol naming such a class. If the class argument is not supplied or is `nil`, open selects one of the following built-in classes according to the direction and element-type arguments:

    excl::character-input-file-stream
    excl::character-output-file-stream
    excl::character-bidirectional-file-stream
    excl::binary-input-file-stream
    excl::binary-output-file-stream
    excl::binary-bidirectional-file-stream

These classes all contain `excl::file-gray-stream` and are variously mixed with

    fundamental-character-input-stream
    fundamental-character-output-stream
    fundamental-binary-input-stream
    fundamental-binary-output-stream

Although the `excl::file-gray-stream` subclasses returned by open are all instantiable, at present they require hidden initialization (for element-type upgrading, buffer allocation, etc.) and therefore they should only be created using open. It is fine to further specialize them, but you are required to create instances of your specializations of these stream classes using the class keyword argument to open rather than by calling make-instance yourself.

On Unix platforms, open will select a terminal-stream subclass rather than a `excl::file-gray-stream` subclass if it detects the argument file is a character special device or a named pipe. See socket.htm for more information about creating sockets in Allegro CL.

## 9.0 Miscellaneous stream functions

The following functions are also defined (or in the case of standard Common Lisp functions, extended).

Generic function | Arguments | Notes
close | stream &key abort | The existing function close is redefined to be a generic function but otherwise behaves as required by standard Common Lisp.
open-stream-p | stream | This function is made generic. A default method is provided by class fundamental-stream which returns true if close has not been called on stream.
streamp | object | The original proposal allowed but did not require these three existing predicates to be implemented as generic functions. In Allegro CL, streamp is not a generic function in order not to impact speed of opencoded type dispatching, but the other two functions are made generic. Normally, the default methods provided by classes fundamental-input-stream and fundamental-output-stream are sufficient.
input-stream-p | stream
output-stream-p | stream
stream-element-type | stream | This existing function is made generic, but otherwise behaves the same. Class fundamental-character-stream provides a default method which returns character.
stream-yes-or-no-p | stream &optional format-string &rest args | The generic function analogue of cl:yes-or-no-p.
stream-y-or-n-p | stream &optional format-string &rest args | The generic function analogue of cl:y-or-n-p.
stream-input-fn | stream | These accessors return the input or output file descriptor for streams that have them, or `nil`. For some streams the input and output file descriptors may be the same.
stream-output-fn | stream

The following function is used by the pretty printer to determine the output width to be used while pretty printing. Actually, the pretty printer uses the first of the following three values that return true.

1. the value of `*print-right-margin*`
2. the value returned by stream-output-width
3. the value of `excl::*default-right-margin*`, an internal variable set when Lisp starts up to the apparent width of `cl:*terminal-io*`, if it can be determined, and to 72 otherwise.

Generic function | Arguments | Notes
stream-output-width | stream | Returns an integer width of stream or `nil`. The default method for this function returns `nil` for all streams.

Copyright (c) 1998-2019, Franz Inc. Oakland, CA., USA. All rights reserved.
This page was not revised from the 8.2 page.
Created 2012.5.30.

ToC | DocOverview | CGDoc | RelNotes | FAQ | Index | PermutedIndex
| Allegro CL version 9.0 Unrevised from 8.2 to 9.0. 8.2 version