ToC | DocOverview | CGDoc | RelNotes | FAQ | Index | PermutedIndex
| Allegro CL version 9.0 Moderate update since 9.0 release. 8.2 version

# Miscellaneous Functionality

This document contains the following sections:

1.0 Introduction
2.0 The fasl reader/writer
3.0 Miscellaneous extensions
4.0 Creating and using pll files
5.0 MD*, SHA*, HMAC, and other message digest support
6.0 Base64 support
7.0 Support for encryption
7.1 Support for Blowfish encryption
7.2 Support for rsa encryption
7.3 Support for the RC4 cipher
8.0 Support for gzip and bzip compression and decompression
9.0 PAM support in Allegro Common Lisp
9.1 PAM conversation functions
10.0 asdf support in Allegro CL
11.0 tar support in Allegro CL
12.0 Coverage analysis in Allegro CL

## 1.0 Introduction

This document describes functionality that does not naturally fit in any of the other overview documents.

## 2.0 The fasl reader/writer

The functions fasl-write and fasl-read provide a mechanism for writing Lisp data and subsequently reading it back into a Lisp image. It can handle many of the common Lisp data types. It can optionally detect circularity and structure sharing in the data and recreate the same topology up to eql-ness of components. The data is written in a binary file format similar to that used for compiled Lisp files, not in ASCII.

Among the advantages of fasl-read and fasl-write over standard Common Lisp read and print functions is that data does not have to be converted to its printed representation prior to being written (that conversion takes a significant amount of time for complex objects). The main disadvantage is that the files (unlike ASCII files with printed representations of Lisp objects) are not at all portable between versions of Lisp (or even between versions of Allegro CL on different platforms or between major releases of Allegro CL).

Not all Lisp objects may be written with fasl-write (and thus are not available to be read by fasl-read). The objects that cannot be written are CLOS objects, streams, stacks, or c-allocated data (cstructs). You can use fasl-open to open a file suitable for numerous fasl-write's. (fasl-read reads all of whatever file is specified to it.)

fasl-read reads the entire contents of its argument file. However, fasl-write, if its argument is a stream rather than a filename, writes data to the stream but does not close it. Thus you can open a stream and do multiple fasl-write's to it. The stream must have `:element-type` `(unsigned-byte 8)`. fasl-open opens an appropriate stream, as shown in the example.

Here is a simple-minded example to show the correct syntax and give an idea of the effect:

    (excl:fasl-write '(a b #1=#:c (#1#) #*1011 #(1 2 3)

    1 1.1 1.1d0 1/2

    #c(1 1) #c(1.5 1.5) "abc")
         "test.fw")
    (excl:fasl-read "test.fw")

    (setq f (fasl-open "test2.fw"))
    (excl:fasl-write '(a b c) f)
    (excl:fasl-write '#1=(a b . #1#) f t)
    (excl:fasl-write '#(1 2 3 4.5) f)
    (close f)

    (excl:fasl-read "test2.fw")

### Application note

fasl-read and fasl-write require the compiler so they are not available in application images without the compiler.

## 3.0 Miscellaneous extensions

The tables describes those extensions to Common Lisp that do not naturally fit elsewhere in the documentation. We only provide brief information in the table. Please follow the link to the documentation page for a full description.

### General functions

Name | Arguments | Notes
excl:dribble-bug | &optional file | This function is an extension of the Common Lisp function dribble. dribble-bug called with the optional file calls several information functions whose output is then placed at the beginning of the dribble file. See also `excl:*dribble-bug-hooks*`.
excl:file-older-p | file-1 file-2 | If file-1 and file-2 both exist, and if file-1 is older than file-2, this function returns t. Otherwise, it returns nil.
excl:gen-sym | x | Similar to cl:gensym except that a symbol is accepted as the optional argument (as well as the standard string or integer), in which case the symbol-name is used. This is useful for programs that are intended for use in either ANSI or modern Lisps, where it is desired to print any gensyms without requiring escapes (see case.htm).
excl:if* | test-form {then then-form+ | thenret} {elseif else-test-form {then else-then-form+ | thenret}}* [else else-form+] | This extension to cl:if allows symbols like then, else, elseif, and thenret in the body allowing a complex number of cases and outcomes to be specified.
excl:named-readtable | name &optional errorp | This function looks up a readtable by name. name must be a symbol, but it is coerced to a keyword (so readtables are named by keywords). setf may be used to associate a name to a readtable. The association can be broken by setting to nil. See also with-named-readtable.
excl:prefixp | prefix sequence | This function returns non-`nil` if sequence starts with prefix, returning the index in sequence where prefix ends (so `(prefixp "foo" "foobar")` returns 3). While this can be done using regular-expression matching (see regexp.htm), this function is easier to use for very simple cases.
excl:uncompile | function-name | If the function function-name was compiled with the compile function (as opposed to having been in a file that was compiled with compile-file and subsequently loaded), then the function is `uncompiled,' i.e. its function definition is replaced by the original interpreted definition. This function will only work when definitions are saved. See the description page for excl:uncompile and the page for `excl:*save-function-lambda-expression*` for details.

### Type predicate functions

Name | Arguments | Notes
excl:bignump | object | These functions, like similar ones in standard Common Lisp return t if object is of the type specified, and nil otherwise.
excl:fixnump
excl:ratiop
excl:single-float-p

## 4.0 Creating and using pll files

A pll file can be used in association with a Lisp image. It contains constant code vectors and strings that can be shared among many Lisp objects. When an image uses a pll file and a function is compiled, the new codevector is compared to codevectors in the pll file. If a match is found, the match is used and no new codevector is allocated. Similarly, if a constant string is specified (with excl:pure-string) and a matching string appears in the pll file, no new string is allocated and the match is used. Strings in a pll file cannot be modified. Attempting to do so causes an error. (Neither can codevectors be modified but there is no user-visible way to modify codevectors as there is with strings.)

Strings and codevectors in a pll file are not also (after being garbage collected) in the Lisp heap. Thus if a string has been successfully purified, it will not be in the heap after a global gc. A total count of strings and codevectors is shown in the output of `(room t)`.

Constant strings in pll files cannot be overwritten (constant strings stored in the heap can be overwitten although doing so is actually illegal). Strings naming Common Lisp symbols are usually stored in pll files, for example. This example shows in that case, the strings cannot be modified:

    cl-user(1): (pll-file)
    #P"/usr/fi/dcl.pll"
    cl-user(2): (nstring-upcase (symbol-name 'car))
    Error: Attempt to store into purespace address #x2d7865e8.
      [condition type: purespace-write-error]

Here we tried to upcase the string naming the symbol `car` (this is a modern Lisp where symbol names are lowercase -- see case.htm). The change failed and signaled purespace-write-error, which is the condition specific to this issue. The next example shows that modifying a constant strings not in the pll file does not signal an error:

    cl-user(88): (setq teststring "this-string-is-likely-not-in-the-pll-file")
    "this-string-is-likely-not-in-the-pll-file"
    cl-user(89): (dotimes (i 4) (gc))
    nil
    cl-user(90): (nstring-upcase teststring)
    "THIS-STRING-IS-LIKELY-NOT-IN-THE-PLL-FILE"
    cl-user(91): teststring
    "THIS-STRING-IS-LIKELY-NOT-IN-THE-PLL-FILE"
    cl-user(92):

The call to nstring-upcase is in fact illegal: you may not modify a constant, but it is an error that does not signal an error unless the string is located in the pll file.

pll files are created with the cvdcvt program described next.

> > cvdcvt

> > Arguments: [`-o outfilename`] [`-u`] [`file1 file2` ]

> > A .pll file, `outfilename`, is created holding all the unique code vectors and strings. If `outfilename` is omitted it defaults to code.blob.

> > If `-u` is specified, then no duplications of strings are done, otherwise for every string that has no lowercase characters in it and at least one uppercase character, a lowercase copy is added to the output file. This is the default and is useful for `set-case-mode`. If no files are specified, stdin is used.

> > .str (string) files and .cvs (code vector) files are combined without redundancies; if two files of the same extension have identical objects, the object from the file specified first to cvdcvt is retained, and the latter object is removed from the output. This allows for files (.cvs files especially) to be arranged by code vectors in order of execution, to provide for locality of reference. Those .cvs files that were produced by training techniques should be placed first in order to have the desired effect.

As said in the description, pll files are built out of cvs files and str files. cvs files are created with sys:write-codevectors and can be created by sys:flush-codevectors. str files are created with record-strings. See also record-code-vectors.

The following functions can be used to associate a pll file with an image, to find out which pll file is used with an image, and to use strings in the pll file.

Name | Arguments | Notes
excl:pll-file | [none] | Return the location of the current .pll file, or nil if there is no .pll file associated with this Lisp.
excl:use-pll-file | name &key (global-gc t) | Associates the current Lisp with the pll file specified by name. It is an error to associate an image already using a pll file with another pll file.
excl:pure-string | x | When not in the body of sys:record-strings, returns a pure-string if there is one identical to the argument and a heap-allocated string if there isn't. When in the body of sys:record-strings, also write the string to the str file being created.

## 5.0 MD*, SHA*, HMAC, and other message digest support

Allegro CL provides various message digest algorithms. Various algorithms are supported, including MD4, MD5, SHA1, SHA224, SHA256, SHA384, SHA512, and RMD160. Rather than have functions for each of these types, there are general functions for them, named digest-* and listed first. HMAC support is available for :md5, :sha1, :sha224, :sha256, :sha384, and :sha512. The functions hmac-init, hmac-update, hmac-string, and hmac-final implement these algorithms.

The MD*, SHA* and RMD160 functions are all are cryptographic hash functions. A hash function takes a long string (or message) of any length as input and produces a fixed-size output. This output is sometimes termed a message digest.

The keyed-hash algorithm HMAC is designed to work with algorithms MD5 and SHA* (for *=1, 224, 256, 384, and 512, to list what Allegro CL currently supports). Keyed hash algorithms work by utilizing an existing hash function (such as MD5 or SHA1) and using a secret "key" as part of the information to hash. If one party sends a message to another party and also includes an HMAC with the message, the receiver (if he/she shares the same secret key) can verify that the message hasn't been altered by running the message through the HMAC functions using the same key. If the two HMACs match, then the message is considered, with high probability, to be unaltered

Many of these function require SSL support (see Secuere Socket Layer (SSL) in socket.htm).

The functions available are:

### General digest functions

- digest-init
- digest-update
- digest-final
- digest-file
- digest-string

These functions are in the `:digest` module loaded with `(require :digest)`.

The following functions for specific encodings are also loaded when the `:digest` module is loaded.

### MD4 functions

- md4-init
- md4-update
- md4-final
- md4-file
- md4-string

### MD5 functions

- md5-init
- md5-update
- md5-final
- md5-file
- md5-string

### SHA1 functions

- sha1-init
- sha1-update
- sha1-final
- sha1-file
- sha1-string

### SHA224 functions

- sha224-init
- sha224-update
- sha224-final
- sha224-file
- sha224-string

### SHA256 functions

- sha256-init
- sha256-update
- sha256-final
- sha256-file
- sha256-string

### SHA384 functions

- sha384-init
- sha384-update
- sha384-final
- sha384-file
- sha384-string

### SHA512 functions

- sha512-init
- sha512-update
- sha512-final
- sha512-file
- sha512-string

### RMD160 functions

- rmd160-init
- rmd160-update
- rmd160-final
- rmd160-file
- rmd160-string

### Related utility functions

- hex-string-to-integer
- integer-to-hex-string
- hex-string-to-usb8-array: converts a hex string to an (unsigned-byte 8) array.
- usb8-array-to-hex-string: converts an (unsigned-byte 8) array to a hex string.

### HMAC functions for MD5 and SHA*

- hmac-init
- hmac-update
- hmac-final
- hmac-string

These functions are in the :hmac module loaded with `(require :hmac)`.

## 6.0 Base64 support

Allegro CL provides support for Base64 encoding within Lisp. Base64 encoding is a 64-bit representation scheme that uses the ASCII characters A-Z, a-z, 0-9, + and /. Since padding could be needed in converting multiples of 8-bits into base64, = characters are used, when necessary, as padding at the end of a converted string. Base64 encoding is described in the RFC2045 document (www.ietf.org/rfc/rfc2045.txt).

These functions provide the Base64 support in Allegro CL:

- integer-to-base64-string, which takes an integer argument and an optional column argument and returns the associated string (with newlines at the column location, if any).
- base64-string-to-integer, which takes a string and returns the associated integer.
- usb8-array-to-base64-string, which takes an (unsigned-byte 8) array argument and a wrap-at-column keyword argument and returns the associated string (with newlines at the wrap-at-column location, if any).
- base64-string-to-usb8-array, which takes a string and returns the associated (unsigned-byte 8) array.
- string-to-base64-string, which takes a string argument and a wrap-at-column keyword argument and returns the associated string (with newlines at the wrap-at-column location, if any).
- base64-string-to-string, which takes a string and returns the associated string.
- base64-encode-stream, which reads bytes from a stream and writes the base64 encoding of those bytes to another stream.
- base64-decode-stream, which reads base64 bytes from a stream and writes the decoded bytes to another stream.

Here are some examples, first using integer-to-base64-string and base64-string-to-integer:

    cl-user(2): (integer-to-base64-string #xfeedfacefeedface)
    "/u36zv7t+s4="
    cl-user(3): (base64-string-to-integer "/u36zv7t+s4=")
    18369614222061337294
    cl-user(4): (format t "~x" *)
    feedfacefeedface
    nil
    cl-user(5):

And now using usb8-array-to-base64-string and base64-string-to-usb8-array:

    ;;  The encoding results may differ between Windows and
    ;;  UNIX/Linux/Mac OS X (the transcript is from a
    ;;  UNIX machine).

    cl-user(5): (setq a (string-to-octets
                         (setq s
                           "
    (defun deep-thought ()
      (sleep (years2secs 7500000))
      42)
    ")
                         :external-format (crlf-base-ef :latin1)))
    #(10 40 100 101 102 117 110 32 100 101 ...)
    cl-user(6): (usb8-array-to-base64-string a)
    "CihkZWZ1biBkZWVwLXRob3VnaHQgKCkKICAoc2xlZXAgKHllYXJz
    MnNlY3MgNzUwMDAwMCkpCiAgNDIpCgA="
    cl-user(7): (base64-string-to-usb8-array *)
    #(10 40 100 101 102 117 110 32 100 101 ...)
    cl-user(8): (setq a2 *)
    #(10 40 100 101 102 117 110 32 100 101 ...)
    cl-user(9): (equalp a a2)
    t
    cl-user(10): (octets-to-string a2 :external-format (crlf-base-ef :latin1))
    "
    (defun deep-thought ()
      (sleep (years2secs 7500000))
      42)
    "
    61
    61
    cl-user(11):

## 7.0 Support for encryption

Allegro CL provides implementations of some publicly available encryption algorithms: RSA (see Section 7.2 Support for rsa encryption) and RC4 (see Section 7.3 Support for the RC4 cipher). Please note that we make no claims about the actual security provided by these encryption schemes. For backward compatibility reasons, we also support blowfish (see Section 7.1 Support for Blowfish encryption), whose use is not recommended because it is no longer considered secure.

## 7.1 Support for Blowfish encryption

Blowfish is an old cipher, designed in 1993. We recommend it not be used, because it is susceptible to various attacks and is no longer considered secure. We maintain support to allow for encrypted material to be decrypted.

The Blowfish algorithm, described on this Wikipedia page (and links from it), is a high speed symmetric cryptographic algorithm (or cipher). The same key is used to encrypt and decrypt the data. Blowfish encrypts blocks of 64 bits (8 octets) at a time. The functions below can automatically pad out the data to encrypt to be a multiple of 8 octets. Blowfish was designed by Bruce Schneier, a leading authority on cryptography and author of the book Applied Cryptography. Schneier writes in his book published in 1996: "I know of no successful cryptanalysis against Blowfish."

- blowfish-encrypt: encrypts an (unsigned-byte 8) array or a Lisp string using a specified key.
- blowfish-decrypt: decrypts a value encrypted with blowfish-encrypt.
- blowfish-init: returns an object that holds the key already processed for doing encryption or decryption, saving time when doing multiple calls to blowfish-encrypt or blowfish-decrypt using the same key. (The object rather than the key is passed to those functions, saving the time needed to calculate the object from the key multiple times.)

Here are some examples of Blowfish encryption and decryption:

    ;; Example 1. string encrypting

    cl-user(12): (blowfish-encrypt "my secret message"
                            :key "my key")
    #(57 27 110 242 191 19 182 150 1 5 ...)
    24
    cl-user(13): (blowfish-decrypt * :key "my key" :string t)
    "my secret message"
    cl-user(14):

    ;; Example 2. (unsigned-byte 8) encrypting:

    ;; Here we allocate an (unsigned-byte 8) array with a size
    ;; that is a multiple of 8 and fill it with data.
    ;; We can do in-place encryption and decryption.
    ;; We specify no padding (since otherwise 8 bytes of padding would
    ;; have to be added and there's no room in this array for that):

    ;; Create our array:

    cl-user(12): (setq aa (make-array 8 :element-type '(unsigned-byte 8)
                            :initial-contents '(2 4 6 8 10 12 14 16)))
    #(2 4 6 8 10 12 14 16)

    ;; Encrypt it in place

    cl-user(13): (blowfish-encrypt aa :key "my key"
                   :in-place t :pad nil)
    #(129 144 108 210 20 227 10 58)
    8

    ;; Verify that it has been modified:

    cl-user(14): aa
    #(129 144 108 210 20 227 10 58)

    ;; Now decrypt in place. Notice how the arguments to
    ;; blowfish-decrypt match those to blowfish-encrypt:

    cl-user(15): (blowfish-decrypt aa :key "my key"
                     :in-place t :pad nil)
    #(2 4 6 8 10 12 14 16)
    cl-user(16):

    ;; And verify that the array is now back to normal:

    cl-user(16): aa
    #(2 4 6 8 10 12 14 16)
    cl-user(17):

    ;; Example 3. use of contexts

    ;; Create context which holds the key processed by
    ;; blowfish to prepare it for encryption/decryption:

    cl-user(21): (setq cc (blowfish-init "my key"))
    #(141 90 172 196 250 88 140 57 179 211 ...)

    ;; Encrypt something using the context:

    cl-user(22): (blowfish-encrypt "my message" :context cc)
    #(75 202 37 143 4 243 181 205 211 126 ...)
    16

    ;; And now decrypt it using the same context
    ;; to show the original string

    cl-user(23): (blowfish-decrypt * :context cc :string t)
    "my message"
    cl-user(24):

It is a common practice to send Blowfish keys to intended recipients using their RSA public keys. See Section 7.2 Support for rsa encryption for information on RSA encryption.

## 7.2 Support for rsa encryption

RSA is a public key cipher named after its inventors: Rivest, Shamir and Adleman. A public key cipher differs from a symmetric cipher like Blowfish (see Section 7.1 Support for Blowfish encryption) in two important ways:

1. There exist two keys: the Public key and the Private key.
2. Different keys are used for encryption and decryption.
3. One of the keys (the Public key) can be made public without making it possible to compute the other key (the Private key).

With RSA you can encrypt with the Public key and decrypt with the Private key or encrypt with the Private key and decrypt with the Public key. Typically one encrypts with the Public key to send a message to the person with the Private key.

RSA has never been proven to be secure. However the obvious way to crack the encryption involves factoring a very large number. There is no published way of factoring a large number that's better than a brute force attempt of trying all possible factors. Thus by making the key big enough you can be sure that it won't be possible to compute the factors by brute force search in a very long time. There may be other ways to crack RSA encryption that are simply not published yet.

One major downside to RSA is that it is roughly 1000 times slower to encrypt and decrypt than a symmetric cipher like Blowfish. As a result people usually use RSA as means of transmitting a key for a symmetric cipher. For example if Alice wants to send Bob a large document securely she'll first go to Bob's web site and copy down his Public RSA key. Then she'll use a random number generator to create a 64 bit blowfish key. She'll encrypt the blowfish key with Bob's Public key and send the result to Bob. Then she'll encrypt her document using Blowfish and the key she generated. Bob will decrypt the first message from Alice using his Private RSA key. That will give him the Blowfish key he'll need to decrypt the second message from Alice.

Because the public key is known to all you have to be careful to not encrypt small values with an RSA public key since that gives you very little security. For example, suppose you decide to encrypt a 4 digit security code using an RSA public key. A person willing to steal your code need only encrypt the values 0000 through 9999 and compare them to your encrypted value to determine what the value encrypted was. If you want to encrypt a 4 digit security code XXXX then it's best to encrypt instead YYYYYYYYXXXX where the Y's are digits chosen randomly.

An RSA key pair consists of three integers: a modulus, a private exponent and a public exponent. The only number that must be kept secret is the private exponent. The public exponent is usually one of a set of common small numbers. The Allegro RSA key generator always chooses 17 as the public exponent.

An RSA key is represented in Allegro as a vector of three values:

1. `t` if this is the public key, `nil` if this is the private key. This value is to help you distinguish one key from the other and is not used in the encryption/decryption code.
2. the modulus value (approximately 1024 bits long).
3. the exponent value.

RSA is a block cipher: a sequence of octets is encrypted at once. The block size isn't fixed but is usually determined by the size of the modulus. In order to encrypt data whose length is not a multiple of the block size padding is done at the end of the value and information about the padding is added to the value. The format of this padding information is not standard among rsa encryption functions, thus you can't expect any function except rsa-decrypt to be able to decrypt a value encrypted with rsa-encrypt.

The functions associated with RSA encryption and decryption are:

- rsa-encrypt: given a value to encrypt and a key, returns the encrypted value.
- rsa-decrypt: given a value returned by rsa-encrypt and a key, return the decrypted value.
- generate-rsa-keys: generates candidate public and private keys. This function takes a long time to complete and is not guaranteed to produce valid keys. Please see the description for details.

Because RSA encryption is resource intensive compared to symetric encoders like Blowfish, it is a common practice to encode using Blowfish and send Blowfish keys to intended recipients using their RSA public keys. See Section 7.1 Support for Blowfish encryption for information on Blowfish encryption.

Here are some examples of Blowfish encryption and decryption:

    ;; A call to generate-rsa-keys, such as the following, can take
    ;; on the order of 10 minutes to complete. The example call
    ;; could have been made with ':verbose t' to get progress
    ;; information as it runs.
    ;;
    ;; The return value is a list of the public and
    ;; private keys, both of which are vectors.  This list
    ;; is made the value of the variable 'keys'.

    cl-user(12): (setq keys (generate-rsa-keys :verbose nil))
    (#(t
       4696616306992156162791359909817969438301590857320651704912\
    5099728659553054438846018512904176959283177314807123575693727\
    9515543419344057970899365859403176313951068268266882944649562\
    1008090347981854919956845970556254842289211552574616675107428\
    9213609596618613446079618857135830766959762009927055865884710\
    796501
       17)
     #(nil
       46966163069921561627913599098179694383015908573206517049125\
    09972865955305443884601851290417695928317731480712357569372795\
    15543419344057970899365859403176313951068268266882944649562100\
    80903479818549199568459705562548422892115525746166751074289213\
    609596618613446079618857135830766959762009927055865884710796501

       38678016645817756634752375727912689491895454119111249334573\
    61154124904369189081436818709755749588026367101763117998307007\
    77506345342165387799477766567321557421166007590816670669406236\
    43773618649101682703783799537062586749082756929026484684965113\
    373035085734095558864183533679809881862774879514342408924287321))

    ;; Here we encrypt with the public key and decrypt with the private key
    ;; we could have encrypted with the private key and decrypted with the
    ;; public key as well.

    cl-user(13): (rsa-encrypt "my secret message" (car keys))
    #(102 136 69 180 180 27 185 63 132 137 ...)

    cl-user(14): (rsa-decrypt * (cadr keys) :string t)
    "my secret message"
    cl-user(15):

## 7.3 Support for the RC4 cipher

RC4 is a stream cipher algorithm. It is used to encrypt streams of data.

The functions available are:

- rc4-set-key
- rc4

These functions are in the `:rc4` module loaded with `(require :rc4)`.

## 8.0 Support for gzip and bzip compression and decompression

The inflate and deflate modules allows you to compress data as it is written to files, and to open streams to files containing gzip compressed data and to uncompress the data while reading the file. To load the modules, evaluate `(require :inflate)` and `(require :deflate)`. Symbols in the modules are in the util.zip package. (There are two modules because they were added at different times.)

The `:deflate` module requires that the libz library be available on your computer (and in the correct version). If you do not have that library or do not have the correct version, deflation will not work. Allegro CL functionality that uses deflation (such as AllegroServe, see aserve/aserve.html) will warn that they cannot compress data but will continue to work without data compression.

The inflation function util.zip:inflate can be applied to instances of the class `util.zip:inflate-stream`.

The various utility functions util.zip:skip-gzip-header, util.zip:skip-gzip-trailer, util.zip:skip-zlib-header, and util.zip:skip-zlib-trailer can be applied to an input stream to position the file position to the correct location for uncompressing and to avoid any trailers.

The deflation functions util.zip:deflate-target-stream, util.zip:deflate-stream-vector, and util.zip:deflate-stream-vector-combined can be applied to instances of the class `util.zip:deflate-stream`.

The value of the variable `sys:*zlib-system-library*` is the name of the gzip library used by the deflate module.

### inflate-stream and related functions

### inflate

### Function

#### Package: util.zip

Arguments: input-stream output-stream

The compressed information from the input-stream is read and the uncompressed information is written to the output-stream.

Both streams must support (unsigned-byte 8) element reading and writing.

### skip-gzip-header

### Function

#### Package: util.zip

Arguments: input-stream

If the input stream is positioned on the header of a gzip'ed file then skip that header. input-stream is not an instance of the class `util.zip:inflate-stream`. It is simply an input stream (opened, for example, with open). The file position must be moved to the beginning of the compressed data before unziping, and only at that point should an instance of `util.zip:inflate-stream` be created (see `util.zip:inflate-stream`).

If the input stream is not positioned on a gzip header then nothing is done.

### skip-gzip-trailer

### Function

#### Package: util.zip

Arguments: input-stream

Skips past the next 8 bytes in input-stream. Note that gzip trailers have no byte-markers to identify them so care should be taken to only call this function after the final data block is read from the stream.

### skip-zlib-header

### Function

#### Package: util.zip

Arguments: input-stream

If input-stream is positioned at the header of a zlib'ed stream, then skip past it, returning the number of bytes read.

If input-stream is not positioned at a zlib header, return `nil`

If input-stream appears to be positioned at a zlib header but turns out to not be, signal an error.

### skip-zlib-trailer

### Function

#### Package: util.zip

Arguments: input-stream

Skips past the next 4 bytes in input-stream. Note that zlib trailers have no byte-markers to identify them so care should be taken to only call this function after the final data block is read from the stream.

### inflate-stream

### Class

#### Package: util.zip

The stream class for instances of files containing comressed gzip'ed data. Instances of this class are suitable as arguments to util.zip:inflate.

To create an instance of this class, do the following:

1. Make a stream (using, e.g. open) with a data source which contains compressed data.
2. If the this file may have a gzip header on it, apply util.zip:skip-gzip-header to the stream.
3. Create an instance of util.zip:inflate-stream by evaluating:

       (make-instance 'inflate-stream :input-handle <stream created in 1>
                      [:compression compress-spec])

   compress-spec can be one of the following three values:

   - `:gzip` (the default): automatically read past gzip header and trailer.
   - `:zlib`: read past zlib header/trailer.
   - `:deflate`: no headers in this stream.
   - `(header-fn trailer-fn)`: users may specify their own functions for skipping past headers and trailers wrapping the deflate-stream. This can be used to support less common encoding methods, such as zip or pkzip, etc. The inflate module does not capture any information from header and trailers, so the custom functions can also be used to capture this information if needed. The function accepts a single argument, which is the input-handle of the inflate-stream. It should return `nil` or the number of bytes read.
   - `nil`: equivalent to the `:deflate` option. No callbacks to utility functions needed.

   The documentation until April, 2016 incorrectly left out the :compress-spec keyword argument. The documentation is now correct.

   The call to make-instance will return a stream which can be read to recover the uncompressed data.

Closing the inflate-stream will also close the stream created in step 1.

### deflate-stream and related functions

To write compressed data, you create a deflate-stream and at that time specify an ultimate taget, which is either a regular stream (perhaps open to a file) or a octet vector (of element type `(unsigned-byte 8)`). You then write data to the deflate-stream and that data is compressed and eventually written to the target. We say eventually because there is a lot of buffering so you do not see data in the target immediately. When you are done, you close the deflate-stream and that causes any remaining data to be written to the target.

Closing the deflate-stream does not close the target if it is a stream. You must close the target stream yourself.

You create deflate-streams with make-instance, as described next.

### deflate-stream

### Class

#### Package: util.zip

The stream class for instances of deflation streams which accepts characters and bytes and causes them to be compressed and sent to a target.

You created a `deflate-stream` with make-instance. You must specify a target when you create a deflate-stream.

Here is a sample make-instance call:

    (make-instance 'deflate-stream :target target-spec
                                   [:compression compress-spec])

target-spec can either another stream, or it can be a vector (that is an actual stream or an actual vector). The stream must be writable. The vector must have element-type (unsigned-byte 8).

compress-spec can be one of the following three values:

- `:gzip` (the default): create a deflate stream with gzip headers.
- `:zlib`: create a deflate stream with zlib headers.
- `:deflate`: create a deflate stream with no headers.

The documentation until April, 2016 incorrectly said there were two choices: `:gzip` and `:deflate`, with `:deflate` creating a stream with zlib headers. The documentation is now correct.

`:gzip` is the preferred format as the result can be uncompressed with the `util.zip:inflate-stream` (be sure to specify `:skip-gzip-header t` to the make-instance creating the inflate-stream or to call util.zip:skip-gzip-header after the input stream is opened). The `:gzip` format output can also be uncompressed with the gunzip program found on Unix.

### A stream as the target-spec

If you pass a stream as the target-spec then as you write characters and bytes to the deflate-stream, the bytes resulting from deflation will be written to the given stream. There is a lot of buffering going on in this stream and the compression library. Therefore you may not see the results in your target-spec stream immediately.

When you close the deflate-stream the last bytes in all the buffers will be sent through deflation and the end of deflation record will be written to the target-spec stream.

Again, the target-spec stream will not be closed when the deflate-stream is closed. It is the callers responsibility to close the stream passed in as the target-spec.

The function deflate-target-stream will return that target-spec stream used by the deflate-stream.

### A vector as the target-spec

Passing a simple vector of type (unsigned-byte 8) as the target-spec is telling the deflate-stream that you wish to collect the deflation result in vectors in the lisp heap. The size of the vector passed is not important. Additional vectors will be created as necessary to hold data as it is written.

Once you have closed the deflate-stream after all data has been written to it, you can retrieve the result with util.zip:deflate-stream-vector or util.zip:deflate-stream-vector-combined, as described in the descriptions of those functions. The deflate-stream-vector-combined combines all results into a new vector if the target vector is not large enough. deflate-stream-vector returns the vectors created by writing to the deflate-stream, along with additional information.

### deflate-target-stream

### Function

#### Package: util.zip

Arguments: deflate-stream

Returns the stream which is the target of deflate-stream which must be a `deflate-stream`. Returns `nil` if deflate-stream has a vector as its target.

### deflate-stream-vector

### Function

#### Package: util.zip

Arguments: deflate-stream

deflate-stream must be a `deflate-stream` whose target is a vector. In that case, this function returns three values:

1. The newest vector created so far by writing to deflate-stream (or the last such vector if deflate-stream is closed).
2. The number of bytes of actual data in the newest vector.
3. A list of previous vectors holding data in reverse order.

For example, if the three returns values are:

        v
        100
        (c b a)

then the deflated result is found by combining in this order:

        all of a
        all of b
        all of c
        the first 100 bytes of v

This function signals an error if deflate-stream has a stream as its target.

### deflate-stream-vector-combined

### Function

#### Package: util.zip

Arguments: deflate-stream

deflate-stream must be a `deflate-stream` whose target is a vector. In that case, this function returns two values:

1. An octet vector.
2. The number of bytes of actual data.

The octet vector is newly created if necessary (if the target vector specified when the `deflate-stream` was created is not large enough to hold the compressed data).

This function signals an error if deflate-stream has a stream as its target.

### Example

Suppose we wish to create a compressed file foo.cl.gz from the following text:

    ;; file foo.cl begin
    (in-package :user)

    (defpackage :foo (:use :cl :excl))
    (defun foo (y) (bar y))

    ;; file foo.cl end

In the following transcript, we use a deflate stream to create foo.cl.gz and then we inflate foo.cl.gz by opening the file, stripping the gzip header, creating an inflate-stream instance, reading the file line by line, closing the inflate-stream, and closing the file. We could also inflate the while file by calling inflate.

    ;; We load the modules and then write a compressed file.
    ;; Once it is written, we read it back uncompressing it.

    cl-user(61) (require :deflate)
    ; [loading messages]
    tcl-user(62) (require :inflate)
    ; [loading messages]
    t
    cl-user(63): (setq myfile (open "foo.cl.gz" :direction :output :if-exists :supersede))
    #<file-simple-stream #P"foo.cl.gz" for output pos 0 @ #x100439f9f2>
    cl-user(64): (setq *df* (make-instance 'deflate-stream :target myfile))
    #<deflate-stream in 0 / out 0 @ #x10043ad922>
    cl-user(65): (format *df* ";; file foo.cl begin~%~%")
    nil
    cl-user(66): (format *df* "(in-package :user)~%~%")
    nil
    cl-user(67): (format *df* "(defpackage :foo (:use :cl :excl))~%")
    nil
    cl-user(68): (format *df* "(defun foo (y) (bar y))~%~%")
    nil
    cl-user(69): (format *df* ";; file foo.cl end")
    nil
    cl-user(70): (close *df*)
    #<deflate-stream in 120 / out 106 @ #x10043ad922>
    cl-user(71): (close myfile)
    t
    cl-user(72): (setq s (open "foo.cl.gz" :direction :input))
    #<file-simple-stream #P"foo.cl.gz" for input pos 0 @ #x1004400732>
    cl-user(73): (util.zip:skip-gzip-header s)
    10
    cl-user(74): (setq is
                  (make-instance 'util.zip:inflate-stream :input-handle s))
    #<inflate-stream
      inflating #<file-simple-stream #P"foo.cl.gz" for input pos 10 @
                  #x1004400732>ef :latin1-base, in: 0, inflated 0, used: 0 of
      @ #x1004417412>
    cl-user(75): (read-line is nil s)
    ";; file foo.cl begin"
    nil
    cl-user(76): (read-line is nil s)
    ""
    nil
    cl-user(77): (read-line is nil s)
    "(in-package :user)"
    nil
    cl-user(78): (read-line is nil s)
    ""
    nil
    cl-user(79): (read-line is nil s)
    "(defpackage :foo (:use :cl :excl))"
    nil
    cl-user(80): (read-line is nil s)
    "(defun foo (y) (bar y))"
    nil
    cl-user(81): (read-line is nil s)
    ""
    nil
    cl-user(82): (read-line is nil s)
    ";; file foo.cl end"
    t
    cl-user(83): (read-line is nil s)
    #<file-simple-stream #P"foo.cl.gz" for input pos 98 @ #x1004400732>
    t
    cl-user(84): (close is) ;; this also closes s.

    t
    cl-user(85):

The source code to the gzip utility is included with the Allegro CL distribution, in [Allegro directory]/src/inflate.cl.

## 9.0 PAM support in Allegro Common Lisp

PAM stands for Pluggable Authentication Modules. It is a flexible mechanism for authenticating users. An Allegro CL module provides a Lisp wrapper around the PAM API on Linux, Solaris, and some other unixlike operating systems. We do not discuss PAM in detail here. See www.kernel.org/pub/linux/libs/pam/FAQ.

PAM is supported on the following platforms:

- FreeBSD
- Linux
- HP/UX (32-bit)
- Mac OS X (32-bit)
- Solaris

PAM is not supported in Allegro CL on the following platforms:

- Windows
- Tru64
- AIX
- HP/UX 64-bit (32-bit is supported)
- Mac OS X 64-bit (32-bit is supported)

The PAM API is is loaded by evaluating `(require :pam)`. Symbols naming functionality are in the util.pam package. Depending on your system configuration, your program may be required to run with 'root' privileges to successfully make use of PAM.

There is one class and several operators defined in the pam module. They are:

- `pam` (class)
- pam-start (function)
- pam-end (generic function)
- set-pam-fail-delay (generic function)
- pam-authenticate (generic function)
- with-pam (macro)

### pam

### Class

#### Package: util.pam

The class of `pam` objects. A pam object is created by pam-start.

### pam-start

### Function

#### Package: util.pam

Arguments: service-name user &key conversation data

This function provides a wrapper around the pam_start(3) PAM library function. service-name should be a string naming the desired PAM service. user may be `nil` or a string.

If conversation is specified, it should be a function (or a symbol naming a function) which will perform the PAM "conversation", when necessary. data is optional user-defined data that will be passed to the conversation function. If conversation is `nil` (the default), a default conversation function will be used. For more information on conversation functions, see Section 9.1 PAM conversation functions.

This function returns a `pam` object on success. You will pass this object to other functions and methods. If pam-start fails, an error is signalled.

### pam-end

### Generic Function

#### Package: util.pam

Arguments: pam &optional status

The default method is analogous to the pam_end(3) PAM library function. pam must be a `pam` object returned by pam-start. If status is specified, it should be an integer. See the pam_end(3) description (in PAM documentation not supplied here) for details on the use of status. If status is `nil`, `excl.osi:*pam-success*` will be used.

This method returns `t` on success, otherwise it signals an error.

### with-pam

### Macro

#### Package: util.pam

Arguments: (var &rest rest) &body body

with-pam is a convenience macro which evaluates body with var bound to the result of calling pam-start with the arguments specified in rest. pam-end will be called when body terminates, either normally or abnormally.

#### Sample use

    (util.pam:with-pam (pam "login" "jimmy")
      (format t "This is the body~%"))

### pam-authenticate

### Generic Function

#### Package: util.pam

Arguments: pam &key flags password

This default method is analogous to the pam_authenticate(3) PAM library function. pam must be a pam object returned by pam-start. If password is specified, it should be a string. It will be used when needed if the default conversation (see pam-start) is used. If flags is specified, it should be an integer.

On success, this function returns `t`.

If the call is not successful, the function returns two values, `nil` and a status value. The status value will be a keyword or an integer. Possible status value keywords are:

- `:auth-err`
- `:cred-insufficient`
- `:authinfo-unavail`
- `:user-unknown`
- `:max-tries`

If pam_authenticate(3) returns an unrecognized status code, it will be returned without being converted to a keyword.

### set-pam-fail-delay

### Generic Function

#### Package: util.pam

Arguments: pam microseconds

This method is used to request a delay of at least the specified number of microseconds (which must be an integer) before returning from an unsuccessful pam-authenticate call. Setting a delay slows down attempts to rapidly try different passwords for an account.

If multiple calls to set-pam-fail-delay are made, the largest requested delay will be used. On some systems, it is possible that the PAM modules themselves may request delays, so you might notice a delay longer than one you requested (and in particular, you might notice a delay even though you hadn't called set-pam-fail-delay).

The actual delay will is computed pseudorandomly and may differ by as much as 25% above or below the maximum requested value.

Regardless of the success or failure of a pam-authenticate call, the delay is set back to 0 before returning from pam-authenticate. This means that you should generally call set-pam-fail-delay before each call to pam-authenticate.

## 9.1 PAM conversation functions

The default conversation function will display prompts and request input from `*terminal-io*`. If the password argument is supplied to pam-authenticate, it is probable that no interaction with `*terminal-io*` will occur at all. However, if your system configuration or application has different requirements, you can provide your own conversation function.

The conversation function will be called by the PAM API when it needs to collect information to move the authentication process along.

If you supply your own conversation function, it should accept two required arguments, and one keyword argument:

Arguments: messages data &key password

messages will be a list of pam-message structures. data will be the same value that was supplied to pam-start. password will be the password that was passed to pam-authenticate (and may possibly be `nil`).

The conversation function should return a list of pam-response structures. The list must have the same length as the messages list. The first entry in the list should be the response that corresponds to the first message. The second entry should correspond to the second message, and so forth. See pam_conv(3) for details.

The pam-message structure has two slots, style and message. The `style` slot will be one of the following keywords `(:prompt-echo-off, :prompt-echo-on, :error, :text)` or an integer (indicating an unrecognized style). `message` will be a string which may be used to prompt the user.

The pam-response structure has two slots, response and code. `response` should be a string with the data requested by the corresponding message. `code` should be an integer (the default is 0). Again, see pam_conv(3) for details.

## 10.0 asdf support in Allegro CL

The popular system definition facility, asdf, is included with Allegro CL. Evaluate `(require :asdf)` to load it into a running Lisp. See [Allegro directory]/code/asdf.readme for more information, `[Allegro directory]/code/asdf.license` for the license, and `[Allegro directory]/src/asdf.lisp` for the source code. asdf documentation can be found on the web at http://constantly.at/lisp/asdf/.

## 11.0 tar support in Allegro CL

Allegro CL provides support for extracting contents from tar files. To use this facility, evaluate

    (require :tar)

The tar functions are named by symbols exported from the util.tar package.

The tar functions take streams (rather than pathnames) as arguments. util.tar:list-tar lists the contents of the tar file. util.tar:extract-tar extracts the contents into a specified directory.

Here are example forms using the tar functions:

    (with-open-file (s "foo.tgz")
      (util.tar:list-tar s :gzip t))
    (with-open-file (s "foo.tgz")
      (util.tar:extract-tar s :gzip t :directory "tmp/"))

### list-tar

### Function

#### Package: unknown

Arguments: stream &key gzip

List, to `*terminal-io*`, the contents of stream, which should be a stream opened to a tar file. If stream is compressed with gzip compression, specify a non-`nil` value for gzip. bzip2 compression is not supported.

### extract-tar

### Function

#### Package: unknown

Arguments: stream &key gzip directory verbose

Extract, to directory, the contents of stream, which should be a stream opened to a tar file. If stream is compressed with gzip compression, specify a non-`nil` value for gzip. bzip2 compression is not supported.

directory defaults to `nil`, which means extract to the current directory (as returned by current-directory).

If verbose is specified non-`nil`, information about what is being done will be printed.

## 12.0 Coverage analysis in Allegro CL

When testing a program, the coverage is a measure of how much of the source code has actually been tested: have all branches of a conditional been taken, have all defined functions been called, have all error handlers been triggered, and so on (coverage is described in Wikipedia here).

Allegro CL has a macro, with-coverage, which executes code and when done, prints information about how well a specified list of functions are excercised. The type of coverage, following the description in the Wikipedia article linked to above, is statement coverage, where "lines of code" is understood to mean Lisp S-expressions.

See the with-coverage page for further details and an example. Note that source file recording must be on for the coverage tool to work.

Copyright (c) 1998-2019, Franz Inc. Oakland, CA., USA. All rights reserved.
This page was not revised from the 8.2 page.
Created 2012.5.30.

ToC | DocOverview | CGDoc | RelNotes | FAQ | Index | PermutedIndex
| Allegro CL version 9.0 Moderate update since 9.0 release. 8.2 version