Allegro Common Lisp FAQ

[Last updated on: 2008-04-17T07:35:59+7:00.]

Administrative Issues

What is the current version of Allegro CL?

The current version of Allegro CL is 8.1 on all supported platforms. This version was released in August, 2007. Or, 8.1 was released 8/1/2007.

How should I report bugs?

What should be included in a bug report is described in the section Reporting Bugs in <Allegro directory>/doc/introduction.htm.

In short, you should include a transcript that demonstrates the failure and error/result seen. Transcripts can be made in a number of ways, such as from an emacs buffer. A logging facility in Allegro CL is also available. Use the function dribble-bug,

To start a session transcript evaluate (dribble-bug "dribble.txt"). This will open the file dribble.txt and write a header containing information about the current lisp environment, which is of help to us when debugging problems. Upon return from this function call, your lisp session is now being logged.

Once finished, evaluate (dribble-bug) (no arguments) to close the dribble bug stream.

The header information is generated by a call to the function print-system-state.

We prefer not to receive screen shots unless there is no other way to transmit info.

Sometimes CL output is not logged in the dribble-bug file. What do I do about this?

If you are using dribble-bug to record output note that dribble-bug should only miss output produced when using the foreign-function interface or running a shell function with run-shell-command and friends, or (because it is produced by C code) information printed by the garbage collector during a garbage collection. Some messages from foreign code may be written directly to Unix stdout or stderr, bypassing Lisp entirely. To circumvent this problem, you can either use an Emacs buffer or the Unix utility "script" to save the entire transaction record.

Is there a mailing list for Allegro CL? How do I sign up?

Yes, there is a mailing list for Allegro CL users. It is accessed through the address allegro-cl@franz.com. To be added to or deleted from this list, send mail to allegro-cl-request@franz.com. Franz periodically makes announcements on this mailing list, but its main purpose is to allow members of the Allegro CL (both Unix and Windows) to communicate with each other, sharing ideas, complaints and code.

Note: the mailing list address used to be allegro-cl@cs.berkeley.edu (and requests went to allegro-cl-request@cs.berkeley.edu). Please note that except for spam filtering, the list is uncensored and membership is open to all. Further note that the accuracy of postings made to the list is not guaranteed.

Is the ACL documentation available on-line?

Yes, at http://www.franz.com/support/documentation/. It is updated from time to time and the instructions for downloading the updated documentation are included in the above link.

Are documentation updates available after Allegro CL is released?

Our documentation is updated online after release as needed. The updated documentation is available on the Franz Inc. Web and versions are available for downloading. See http://www.franz.com/support/documentation/ for both online documentation and downloadable updates.

Does Allegro CL run on operating system X?

Our policy of support of Allegro CL on specific operating system versions is as follows:

For each released version of Allegro CL, we compose a list of current operating system levels on which we have tested and on which we know our product will work. This list is also located in installation.htm in the section Installation sizes and supported Operating System versions.

Because Allegro CL runs on so many different operating systems, and because some of these operating systems have almost infinite combinatorial possibilities, including patches and/or user-modification, we do not attempt to certify Allegro CL on specific operating system versions. Instead, we rely on the tendency of the operating system vendors to maintain upward compatibility, and we usually build on the lowest operating system version for which we have listed support. As long as the operating system is truly upward compatible, Allegro CL should work on the newer versions, and our advice is to try it out. But we also recommend strongly that you run such tests on a machine that is not part of your production process, in case things don't go well.

What is the best question to ask of Franz Inc as to my particular operating system and Allegro CL?

If the operating system you are interested in running Allegro CL on is the same lineage and the underlying architecture is the same as one which we have already listed as supported, the best question to ask us is

• Have you had any compatibility issues with running Allegro CL on Operating system XYZ?

We can answer this question "yes" or "no", which will tell you as the customer whether we have any negative experiences with the operating system version. (The information here may also provide the answer.) If the operating system is very new, the answer to this question might more likely be "no", because the assumption is that the new version is compatible, unless and until it proves otherwise. Other questions that you might ask, if you know that your operating system is very new, are

• Do you know of customers who are successfully running Allegro CL on Operating system XYZ?

• Do you have plans to support Allegro CL on Operating System XYZ?

The fundamental issue with Allegro CL working on a new operating system versions is whether the operating system has any fundamental issues that break Allegro CL. Typically, when the operating system is a natural progression from one which we currently support, we will take specific steps to resolve incompatibilities. Incompatibilities most often occur when an operating system vendor changes the signal-handling interface, usually to be more Posix compliant. The change can break Allegro CL. This has happened in the last several years to Linux, LinuxPPC, FreeBSD, and MacOSX, in that time order.

Patches

How do I install patches?

The various processes for installing patches are described at http://www.franz.com/support/patches/

Is there a list of available patches?

Yes. In fact, even if you are using sys:update-allegro to download patches, we highly recommend viewing this file periodically so that you are aware of what each patch being downloaded is affecting.

Here are the LOG files for each currently supported platform

• http://www.franz.com/support/patches/log/8.1/index.lhtml

• http://www.franz.com/support/patches/log/8.0/index.lhtml

• http://www.franz.com/support/patches/log/7.0/index.lhtml

We also provide RSS Feeds to Patch releases for current Allegro CL Versions, and both technical and general announcements. You can check the patch LOG page above periodically or subscribe to one of the RSS feeds to be informed automatically.

How do I manually download patches if I am unable to use (sys:update-allegro)?

We maintain a publically accessible FTP site from which users can download all available patches. Please visit: ftp://ftp.franz.com/pub/patches/8.1/. From the parent directory, one can also access patches for past versions of Allegro CL.

This directory contains a README file with instructions for manually downloading patches toward the bottom of the file.

As patches become available, the files in this directory will be updated, so it is a good idea to be aware of when patches are released. The FAQ item here provides information on how you might do so.

For a general explanation on how to find and load the patches for your version of Allegro CL see http://www.franz.com/support/patches/.

How can I install Allegro CL patches on Windows Vista?

Updating patches is a two-step process. First, a user typically calls the function (sys:update-allegro) to download all available patches, exits all lisps, then runs update.exe. Both of these operations result in the creation or modification of files in the Allegro directory. Windows Vista has a security model that disallows installation of files into c:\Program Files\ if you do not have Administrator priviledges, which is where Allegro CL is installed by default. One solution, then, is to log in as an administrator when performing an update of your Allegro installation. This can be tedious and inconvenient. The second option is to use the "Run as administrator" right-menu option to temporarily elevate your priviledges.

To install patches first navigate to the Allegro CL start menu and right-click on the image you'd like to start. Select Run as administrator. When you select Run as administrator you will need to answer in the affirmative to the User Account Control dialog boxes Vista pops up. This will enable you to install the patch files into c:\Program Files\. When the image starts, call the function sys:update-allegro and all current patches should be successfully downloaded.

To rebuild your images with your current set of patches you must start the cmd.exe program with elevated priviledges, as explained above (that is, navigate to it, bring up the right-click menu, etc.). Then, you should cd to your Allegro directory and run update.exe. If there are no errors during the update process, you have successfully installed the current set of patches.

Express Edition installation and license file issues

[Express] [Windows] Can I install the Express Edition if I do not have access to the internet?

No. Our Windows installer requires access to the Franz Inc. web site (www.franz.com) to be able to retrieve your license file. There is no other way to get the license file. If your goal for installation of the Express Edition is the evaluation of our technology, we will be happy to provide you with an evaluation version of our Professional or Enterprise product, which can be installed without internet access, though email address for the evaluation license file is necessary. Please contact info@franz.com to request your evaluation.

[Express] [Windows] The Express does not install--what can I do?

As of version 8.x, getting a license file is done by the Express installer. There is no other way to get a license file. Here are the most common causes of problems for the installer retrieving a license file:

• a proxy is required to retrieve the license file: if you require a proxy to access web sites, then you will need to configure the Allegro installer to use it. On the Allegro CL Download Location dialog there is a Configure HTTP Proxy button. Click on it and fill in the HTTP Proxy with the host name of your proxy server, and Port with the port on that server on which the proxy listens. If your proxy server requires authentication, then fill in the Username and Password fields with your user name and password. (The user name and password are not transmitted to us, they are only used to access our website.)

• antivirus software prevents the installer from running: we have seen cases where antivirus software detected our installer has a virus. If you downloaded the installer from our web site, we are sure that it does not contain a virus, as we maintain up-to-date virus scanners on all development machines.

• security software prevents retrieval of the license file: some antivirus or security suites block some outgoing HTTP connections, like that used to retrieve the Express installer and license file. Known issues:

  • NOD32: The IMON module of this antivirus software will likely cause problems for retrieval of a license file. For some users, merely disabling the IMON module will allow our installer to function properly. In others, uninstallation of NOD32 was required.

• malformed entries in the PATH environment variable can cause the installer to abort: it is known, for example, that an errant double backslash will cause a License Retrieval Error with error text about an illegal namestring:

  

[Express] [Linux/FreeBSD/Mac OS X] How do I install the license file?

See How do I install the license file below.

[Express] [Windows] Is my antivirus software correct that the Express Edition is a virus?

No. If you downloaded the software from our web site, we believe your antivirus software is giving you what is called a false positive. OfficeScan from Trend Micro, and other Trend Micro antivirus products have been known at various times to give false positives on Allegro products (not just our Express Edition).

[Express] How long can I use the Express Edition?

You can use it until the next release of the Express Edition has been out for some time. We usually give Express users 6 months or more to migrate to the new version.

[Express] Does the Express Edition expire?

Not exactly. The license file does require renewal, but that process is simple, even automatic on Windows. When a new version of Allegro CL is released, we do halt the renewal of Express licenses for older versions and you will be asked to install the most recent Express.

[Express] How do I update my license file?

Periodically, every 30 days, your license file is updated. You will be asked to update it when it is expired. When it is time to update the license file, you will be asked if you want to update the license file. If you say yes, your license file will be automatically updated. If you require a proxy to access web sites, see this entry.

[Express] While updating my license file I get an error: "Unknown host: lfs.franz.com". What do I do?

The process of updating the license file must communicate with our web site, and in order for this to work you must be online when the license file is updated. If you are connected to the internet and you get this error, there may be a problem with hostname lookups on your system. Make sure your DNS server is properly configured, asking your Network Administrator for assistance if necessary. If your DNS server is set up correctly, you should be able to:

• use a browser to access http://www.franz.com, and

• run ping lfs.franz.com successfully from a command/DOS prompt.

[Express] How can I update my license file if I require a proxy to access web sites?

This is the only case in which updating the license file requires manual intervention. To update your license file, you should run the newlicense program contained in the Allegro directory, like this:

newlicense -p proxyhost:proxyport
    

where proxyhost and proxyport are the hostname and port on which your proxy server resides. If your proxy requires a username and password, then the form would be:

newlicense -p proxyhost:proxyport -P username:password
    

[Express] Why do I get a "Connection Refused" or "Connection timed out" error when updating my license file?

NOTE: Windows Express users cannot use this method because the license file is not stored based on your email address.

This problem is most likely to occur because of your firewall or proxy configuration. If you are able to access http://www.franz.com via a browser, try to use the http://www.franz.com/lfs/update-trial-license page for updating as described here.

If you continue to have problems, or newlicense reports any errors, please send us email at support@franz.com describing your problem. Please send us the contents of devel.lic in your message.

[Express] How do I update my license file if I do not have internet access?

Try using http://www.franz.com/lfs/update-trial-license to update your license file instead of newlicense.

[Express] When I try to run Allegro CL, a message says my license has expired, but when I run newlicense, it says it is too early to renew. Why might this happen?

This can happen when the date on your computer is incorrectly set to a date in the future later than the license expiration date. Allegro CL will not start because it thinks the license has expired. You cannot renew because the Franz Inc. computers (which are presumably set to the correct date) determine it is not time to renew. You can check the date on your computer by looking at the Date/Time entry on the Control Panel. Please consult your Windows documentation for information on resetting the date on your computer.

[Express] How do I build mlisp, alisp, or allegro images?

Windows: paste this into the Debug window to build one of mlisp.exe or alisp.exe:

;; mlisp:
(progn
  (build-lisp-image "sys:mlisp.dxl" :case-mode :case-sensitive-lower
		    :include-ide nil :restart-app-function nil
		    :restart-init-function nil)
  (when (probe-file "sys:mlisp.exe") (delete-file "sys:mlisp.exe"))
  (sys:copy-file "sys:allegro-ansi.exe" "sys:mlisp.exe"))

;; alisp:
(progn
  (build-lisp-image "sys:alisp.dxl" :case-mode :case-insensitive-upper
		    :include-ide nil :restart-app-function nil
		    :restart-init-function nil)
  (when (probe-file "sys:alisp.exe") (delete-file "sys:alisp.exe"))
  (sys:copy-file "sys:allegro-ansi.exe" "sys:alisp.exe"))

;; allegro:
(progn
  (build-lisp-image "sys:allegro.dxl" :case-mode :case-sensitive-lower)
  (when (probe-file "sys:allegro.exe") (delete-file "sys:allegro.exe"))
  (sys:copy-file "sys:allegro-ansi.exe" "sys:allegro.exe"))
    

UNIX: evaluate the following form to build mlisp image from a running alisp:

(progn
  (build-lisp-image "sys:mlisp.dxl" :case-mode :case-sensitive-lower
		    :include-ide nil :restart-app-function nil)
  (when (probe-file "sys:mlisp") (delete-file "sys:mlisp"))
  (sys:copy-file "sys:alisp" "sys:mlisp"))
    

Evaluating any of the above forms does not add a menu item to the Start Menu. To run the resulting image, you will need to run the executable created in each form.

[Express] Is Lisp in a box available with the most recent Express Edition?

Yes, here:

• http://www.gigamonkeys.com/book/lispbox/

• http://www.franz.com/PracticalCommonLisp/

[Express] After Installation on Windows, I get a "license not found" error. How can I get my license file?

When downloading Allegro CL Express Edition for Windows, a single executable, Allegro81installer.exe, downloads, installs, and retrieves the license file that will allow the software to work. If you did not run the Allegro81installer.exe to install the Express Edition, and instead ran acl81_express.exe, you performed a partial installation and must uninstall, then reinstall using the proper executable.

If there is a problem in the license retrieval step, please check carefully for any dialogs that indicate failure during the installation process. These messages will provide clues to tracking down various network configuration issues that are covered in other sections of the Express Edition FAQ.

If the operating system requested a reboot during the installation process and you opted to Reboot Now, please reinstall and instead choose the Restart Later option.

Franz Inc. is unable to generate licenses manually for the Express Edition on Windows. A License must be retrieved via successful completion of the installation program.

If the use of the Express Edition is intended as a corporate evaluation of the software, please consider emailing our sales department at sales@franz.com to find out about other ways in which you might demo our software.

Professional/Enterprise Edition installation and license file issues

How do I install the license file?

You should have received an email containing a URL from which you can download your license file. (Click here if you haven't received that email or if you have mislaid it. Click here if you cannot access the URL.)

1. The email we sent you in response to your download order contains a customized URL to your license file. Click on that URL or copy it into a browser to retrieve your license file.

2. Save the text to a file called devel.lic. (the name is "devel" and the file type or extension is "lic".) On Linux, this is usually not a problem, but because Windows usually hides file extensions some users have problems getting Windows to save the file correctly. When you view this in the Windows Explorer, its file type must show as lic, not txt or text or anything else. Therefore, you must be careful to save this file so that its type is lic and its contents are plain text. Each version of Windows has a slightly different method of identifying and listing file types. Please refer to the "Windows Users" section to identify how to do this on your individual machine. In some instances, Windows will automatically save the file as an HTML file (devel.lic.htm), adding an HTML header and footer. These headers will prevent the license file from working.

3. If you used the "lost license" URL to obtain your license file, be sure that you save only the license file portion of the email into devel.lic. If you save the entire email message, MIME headers and all, it will prevent the license file from working.

4. Put the devel.lic file into the Allegro directory.

I have misplaced the email telling me the URL from which I can download your license?

NOTE: Windows Express users cannot use this method because the license file is not stored based on your email address.

We can send the license to you by email. (The link is to http://www.franz.com/lfs/lostlicense and on that page, you are asked for the email address you entered when you requested the license; entering the email address and clicking the Submit button will cause your license file(s) to be emailed to you. Please wait after clicking Submit for the message saying the request was successful. If no license associated with the email address is in the database, the message `No licenses found' is displayed. If that happens, send a message to support@franz.com. Be sure to tell us the email address you used in the body of the message.)

I can not access the URL for retrieving my license. What should I do?

Typically, this problem occurs because your system is behind a firewall or you must use a proxy server. Our license file server rarely experiences any significant downtime.

You should make sure that you should not be using a proxy server to access sites outside of your network and that your browser is configured appropriately. If you do not know if you should be using a proxy server, or don't know what one is, you should check with your Network Administrator for assistance. If your system does not require a proxy server, then perhaps you are behind a firewall that does not give access to certain ports.

If the proxy server is not the problem, please email Franz Inc. at support@franz.com stating that you are unable to access the URL for retrieving your license (include the URL), and we will have the license(s) sent to the email under which you have registered.

I still have problems with my license file. Can I contact Franz Inc. for assistance?

Yes. Questions or problems should be sent to support@franz.com. Please be sure to include the following information:

• what happened, and what steps you took;

• a copy of your current license (devel.lic) file;

• a copy of any error messages that you received; and

• your current email address and daytime telephone number

Using Allegro Common Lisp

[Express Edition] Can I use the SSL interface in Allegro CL Express Edition?

No. While the SSL module fasl file was included in the Express, it was not intended that the SSL interface be available in the Express Edition. If you have the need for an SSL interface, we gladly urge you to contact us at info@franz.com for an evaluation of our Professional or Enterprise Editions.

Why doesn't make-pathname merge the given :directory component with the directory component in :defaults argument?

Section 19.4.4 of the ANSI spec says:

After the components supplied explicitly by host, device, directory, name, type, and version are filled in, the merging rules used by merge-pathnames are used to fill in any unsupplied components from the defaults supplied by defaults.

unsupplied is the crucial word here. By specifying a :directory argument you have supplied the directory component, and the directory component of the :defaults argument is not used. Even specifying :directory nil explicitly supplies a directory component of nil, and this will be treated differently from unsupplied.

I am getting stack overflows and occasional Lisp failure when I sort on large arrays. Why and what can I do?

Here is a transcript showing a stack overflow. Note that the array has one million (10^6) elements.

USER(1): (setq pippo (make-array 1000000 :initial-element 0)) 
#(0 0 0 0 0 0 0 0 0 0 ...)
USER(2): (sort pippo #'<)
Error: Stack overflow (signal 1000)
[condition type: SYNCHRONOUS-OPERATING-SYSTEM-SIGNAL]

Restart actions (select using :continue):
 0: continue computation
 1: Return to Top Level (an "abort" restart)
[1c] USER(3): :pop
=================^^^^
USER(4): (sort pippo #'<)
#(0 0 0 0 0 0 0 0 0 0 ...)
USER(5): 
    

Here the computation continues and Lisp exits with a segmentation violation:

USER(1): (setq pippo (make-array 1000000 :initial-element 0))
#(0 0 0 0 0 0 0 0 0 0 ...)
USER(2): (sort pippo #'<)
Error: Stack overflow (signal 1000)
 [condition type: SYNCHRONOUS-OPERATING-SYSTEM-SIGNAL]

 Restart actions (select using :continue):
 0: continue computation
 1: Return to Top Level (an "abort" restart)
[1c] USER(3): (sort pippo #'<)
Segmentation fault (core dumped)
%
    

The stack overflow occurs because a large array is being stack-allocated to perform the sort. The size of the array is architecture dependent; Windows platforms only allocate up to 4 Kbyte arrays on the stack, and normally heap allocate any larger arrays needed, while Unix platforms attempt to allocate 4 Mbyte arrays on the stack. On any architecture, the strategy is programmable; as described below.

When the above error occurs, there are several things that can be done.

1. Instead of popping out of the break loop as in the example above, just continue. The stack overflow automatically reduces the stack cushion (see documentation for sys:stack-cushion and sys:set-stack-cushion), so continuing should allow further execution.

2. On Unix platforms only, a csh can be run and the limit command used to set the stack limit to something larger than it currently is. We recommend at least 8192 Kbytes (8 megabytes), but if that is not enough, more can be allocated.

3. Change the sort strategy (documented below). The Allegro CL implementation of the sort function tries to allocate a temporary array on the stack if possible, so that it does not need to do so on the heap. If this strategy is not acceptable or convenient, change the strategy to either allocate from the heap or to use a pre-existing user supplied array.

Just continuing usually works as does, usually, clearing stack with a :reset and retrying. Note, as the second example above shows, trying to redo the sort command in the error prompt (that is, without clearing the error) can result in an abnormal exit from the lisp (Segmentation fault (core dumped) ).

This is an unfortunate hole in our stack-overflow detection strategy; Stack overflow is normally detected for every function call, and enough "slop" is allowed for so that functions that allocate an average amount of stack will not cause a hard stack overflow. But if the function allocates large stack objects (such as large temporary vectors) then the jump in stack usage is too much to detect by either the stack cushion or the hardware overflow detection, and stack-overflow death occurs. We hope to guard against such overflow death in some future version of Allegro CL.

Sort Strategy:

You can tell the system whether to try to stack-allocate things to be sorted. From the documentation in the source code:

;; excl::*simple-vector-sort-strategy*:
;;
;; The sort strategy can be one of three types:
;; :stack - try to allocate stack space for the temp sort; this
;; works easily for 1k elements (4 kbytes), and (on
;; Unix platforms only) for up to 1m elements (4 mbytes)
;; if there is enough stack allocated by the os; more
;; than 1 m elements cause a new svector to be allocated.
;; :alloc - Allocate an svector of size equal to the vector to sort.
;; a new one is allocated each time.
;; <vector> - must be a simple-vector of type t of at least as many
;; elements as are being sorted. During the sort, the global
;; is reset to :alloc so that sort is re-entrant.

(defvar excl::*simple-vector-sort-strategy* :stack)
    

I have set the stack cushion (see sys:set-stack-cushion and sys:stack-cushion) to a reasonable value, but the soft stack limit is not being detected, and I get a lisp death instead. Why is that?

The stack-cushion (see sys:set-stack-cushion and sys:stack-cushion) is detected in "symbol trampoline", a short piece of code that is used when one Lisp function calls another. It is meant to flag normal situations where stack is growing too quickly, and to signal a condition before a hard stack-size limit is reached.

There are several possible situations where the stack-overflow is not detected by this mechanism, and careful thought must be given as to how to handle it:

1. A lisp function may allocate a very large stack size, due to either a large number of variables or due to large stack-allocated arrays or lists. If the amount that the function allocates is larger than the difference between the hard stack limit and the soft stack limit set up by the stack cushion, then there will be no chance for the Lisp to signal the condition before the hard limit is reached. The only way to work around this problem is to be sure that there is sufficient stack-cushion for the worst-case function to allocate its needed stack.

2. A Lisp function might call itself recursively, which on some architectures generates a fast call to location 0 of the same function. The fast call causes the symbol trampoline to be bypassed, thus causing the stack overflow detection to also be bypassed. The workaround is to declare the function calling itself as notinline within its own body. This will result in slightly slower code generation, but overflows would then be detected. Example:

    (defun call-me ( ... )
      (declare (notinline call-me))
      ...
      (call-me ...) ... )
    

1. A non-lisp thread may be called, at which time there is no way to limit the stack on some machines. There is no workaround for this problem, other than to reduce one's dependence on non-lisp code.

Why does it take so long to load a file that interns several thousand symbols in a package?

A package has an associated hashtable for the names of symbols in the package. When the size of a package is not specified at creation time, a default hashtable is used. Its initial size is small, allowing for 10 entries, and it tends to grow slowly, growing about 20% each time growth is necessary. Those values are reasonable for most uses, but if you know that a package will have many more symbols, particularly if they will be all created at roughly the same time (as when reading a file that interns thousands of symbols), you should specify the :size keyword argument to defpackage appropriately when creating the package. Thus, if you know the package will eventually have about 4000 symbols, define it with a form like this:

(defpackage :foo (:size 4000) (:use :cl :excl))
    

Heap placement issues

How can I specify very large heap sizes for 64-bit versions of Lisp?

Something like this will not work of some platforms (64-bit Windows, for example):

(build-lisp-image "new.dxl" :lisp-heap-size (* 1024 1024 1025 15))
    

Instead of allocating a 15 GB heap, it will allocate something significantly smaller. This is caused because the integer resulting from the multiplication (16,121,856,000) requires more than 32-bits to represent and the ascii to integer conversions on some operating systems do not handle more than 32-bits.

Instead, do this:

(build-lisp-image "new.dxl" :lisp-heap-size "15375m")
    

Sometimes Allegro CL, particularly with large images, fail totally with a bus error or a segv. Why might this be happening?

In large images, this is occasionally a sign that your system has run out of virtual memory. This can occur on platforms that perform lazy swap allocation, which makes possible the overcommitting of virtual memory. Heap relocation or resizing will not help resolve this issue. Instead, you should:

• Have your system administrator increase the available system resources.

• Use the Runtime Analyzer to determine what is causing the large allocations and resultant heap growth.

• Tune the Garbage Collector to better meet the needs of your application.

Sometimes Allegro CL, particularly with large images, run out of memory with a storage-condition. Why might this be happening?

The most common cause of this problem is that you've run out of address space for the lisp heap. The first question to ask yourself, as a developer, is: Do I expect my application to consume as much memory at is it?

If the question is no, you should determine the cause of the unexpected allocations and resultant heap growth. The Space Analyzer can help you do so.

If the answer is yes, then it may be that you need to adjust the locations of the heaps used by Allegro CL. Choosing an adequate location in which to map the Lisp and foreign (C) heaps in a running Lisp image is complex. We refer to these problems collectively as the heap placement problem. While these problems are not in fact new, they are only triggered when the Lisp image is large (typically greater than 500 Mbytes).

Continue reading the questions below for advice on how to proceed.

How is heap placement determined and what can go wrong?

When Allegro CL starts up, space must be found for the following:

• The Lisp heap. This is where Lisp data is stored. This heap may be moved or shrunk in order to fit into available memory at startup.

• The C heap. This is where space allocated by aclmalloc is located, along with the value of certain C variables. Note that C heap is an unfortunate name because it is not directly related to C and is, in fact, directly managed by lisp. The C heap must always be allocated at an address higher than the Lisp heap. This heap is static and will not be relocated if necessary at startup. This is due to a requirement to maintain the accuracy of pointers into the C heap across calls to dumplisp.

• Shared libraries. These are the .so (on most UNIX machines), .sl (on 32-bit HP-UX), .dll (on Windows), and .dylib (on Mac OS X) files built with the system.

• pll file: the `pure lisp library' file contains constant data (such as strings and code vectors) to be used by Allegro CL when running. The name refers to the file type. Images can be built to use a pll file or not. See Creating and using pll files in miscellaneous.htm.

• The stack.

If you use 32 bit addressing (as Allegro CL does on most platforms), there are potentially 4 GB of address space. However, most operating systems only allow 31 bit addresses, so only 2 GB are really available. Locations near address 0 (the bottom) are usually reserved by the operating system. Therefore, Allegro CL usually tries to start at some OS-dependent location, typically around 0x20000000, so there is potentially a 1.6 GB block above that. This is plenty for an application that uses less than, say, 0.5 GB, but as the application grows to, say, 1.5 GB, finding enough space becomes problematic. Assuming no intervening shared libraries or other allocated regions in the process address space, you can determine your maximum lisp heap in an image by subtracting the lisp-heap-start (lisp base) from the c-heap-start (c base).

Lisp is given an idea of how much heap space it will need to operate via the lisp-heap-size argument to build-lisp-image. This value does not describe the extent of the heap in use, rather it indicates a guaranteed size to which the heap should be able to grow. When building large images on 32-bit platforms, process address space is at a premium and will typically be exhausted long before physical RAM or Virtual Memory is consumed. It is important, therefore, to find and claim a suitable block of address space for application needs.

The claiming--via mmap(), specifically--of reserved growth space for the heap has differing behavior under different operating systems. For systems that support the MAP_NORESERVE flag or perform lazy allocation, a lisp-heap-size'd block of address space will be claimed without any resultant mapping to virtual memory (until it is actually used). For other platforms a request for address space will also claim an equivalent amount of virtual memory. Currently, out of all supported Allegro CL platforms, only Tru64 and HP-UX 10.20 meet this latter criteria.

So, what might go wrong when Allegro CL starts up? The following might be problems:

• On platforms (HP-UX 10.20 and Tru64) that do not support a reserve/commit distinction, there may not be sufficient swap space to accommodate the requested Lisp heap size.}

• Shared libraries may not be mapped at their desired locations. The worst offender in this regard are apps run under the Windows operating systems. In some cases when a .dll cannot be mapped where it wants to be it will not work properly.

• Even if there is space to map everything, the first try at mapping may fail and everything may need to be remapped. If the relocation is not done properly, this can cause Lisp to fail.

Even though this problem may affect any user of Allegro CL or an Allegro CL application, it is mostly a problem with developers of programs which will be distributed to a user base. Users who have an Allegro CL distribution and use a particular machine can, perhaps with trial and error (and perhaps with assistance from Franz Inc.), figure out how to build an image that will work on that machine. But VAR's, say, who are preparing a distribution requiring large images which will be sent to many customers, each of whom may have a different machine configuration and different programs running, may find it difficult to produce a single image suitable for all potential users on a particular platform.

Programmers can affect heap placements using these arguments to build-lisp-image:

• lisp-heap-start

• lisp-heap-size

• c-heap-start

• c-heap-size

Improvements in heap location management made available starting in release 5.0.1 make the successful mapping of large images more likely, and make the providing of one image for all users on a particular platform easier to solve. But, any application that uses most of the available address space is in danger of running into machine constraints which make the application fail.

How does Lisp start up, in terms of shared-library linking and loading?

This is a complicated answer. We start with some terminology:

• Shared-library: A program unit that can be linked or loaded into a program, which usually has a .dll extension for Windows, a .sl extension for 32-bit HP-UX, and a .so extension for all other UNIX systems. Shared libraries come in three flavors:

  1. The ACL shared library: This shared-library holds the base ACL system, and is sometimes known by the term "acldll". On Windows it is known as aclxxx.dll, and on UNIX it is called libaclxxx.ext where xxx is a version number and .ext is either .sl or .so.

  2. System libraries: shared-libraries that are pre-linked into either the ACL shared-library or the executable that loads the ACL library, or any shared-library on which a system library is dependent. This is a broad definition of system library, and can include any user-supplied library that has been linked into the executable.

  3. User loaded libraries: These are libraries that are loaded with the Common Lisp LOAD function, but not pre-linked as are system libraries.

• Executable: a program that either links or loads and invokes the ACL shared library. We supply several executables in the distribution, one called mlisp.exe on Windows and mlisp on unix. However, any program can be built that does the equivalent (see on Windows, for example, dll.htm). The executable loads the ACL shared library, and invokes it by getting and executing the lisp_init() function. (You can provide your own main() that calls lisp_init(), see main.htm).

• Heap file: a file with a .dxl extension (also called an image file) that holds the two Lisps heaps. One is the C heap, which is not directly C related, but which holds non-Lisp data including C variable values and data allocated by aclmalloc. The other heap is the Lisp heap, which holds all mutable Lisp data.

• Pure Lisp library: A file with a .pll extension, which may be optionally loaded into a Lisp process and which contains read-only Lisp data.

• Link: A process of attaching shared-libraries by name to an executable, usually by a linker program or by a compiler that automatically invokes a linker. Shared libraries and their symbols are not usually actually included into a program when linked, but references by name are made and the linked library is required to be available before the executable can start.

• Load: A process by which a shared-library is dynamically mapped into the memory space of an already-running executable. The functions to call to load a shared-library are LoadLibrary() on Windows, shl_load() on HP-UX with 32-bit Lisps, and dlopen() on other unix systems. The load Common Lisp function of Allegro CL calls one of these functions when it sees that the file being loaded is a shared-library.

• Bind: Attaching functions within a shared-library to allow those functions to be called. The ff:def-foreign-call function in Allegro CL creates a binding location that is automatically updated when shared-libraries are loaded that supply the functions. The executable must bind the symbol lisp_init in the ACL shared library in order to call it. Binding is done by getting the address of the function's code, by GetProcAddress() in Windows, shl_findsym() on HP-UX, and dlsym() on other unix systems.

• Invoke: The process of calling a function which has been bound. C invokes a function that has been bound by using a "pointer-to-function" construct. Lisp uses ff:def-foreign-call to bind a C function in such a way that it looks like a Lisp function.

• Committed area: An area of memory that is mapped in and which consumes virtual memory, one page of virtual memory for each page of address space. The memory might be physical memory (RAM), or it can be swap memory on disk, or it might even be a specific file, also on disk. The Committed area of the Lisp heap is usually the area bounded by the lisp-heap-start and the "Top" address in the room display.

• Reserved area: An address range of memory is reserved when no other program unit has the right to map anything into that range, but the address range is not committed by actual swap space. This is the area requested by the :lisp-heap-size argument to build-lisp-image (see building-images.htm). It is carried through all dumplisps. Although the committed area may eventually grown beyond this value, the space for the reserved area is only allocated if there is enough space for it. The reason for having a reserved area, instead of simply allocating memory that the operating system will give, is because as the Lisp heap grows, its addresses must grow monotonically increasing; new spaces must be always at higher addresses. (Note: HP-UX 10.20 and Tru64 systems do not provide the concept of reserved areas; instead, all memory that is mapped is immediately committed to virtual memory as well. On these systems, one should estimate the lisp-heap-size more accurately, since more swap is consumed for larger lisp-heap-sizes).

The Startup Process:

1. The operating system starts up the executable. Before the executable is able to start running, all system libraries must be loaded into memory and available. Various operating systems do this in various ways, using implementation dependent algorithms to place the libraries into memory. Also, whether all symbols are bound at the time of the library placement or whether the symbols are bound lazily (i.e., when needed only) is OS dependent; some systems may provide both.

2. The executable begins to run. It may perform any operations it wants to do, including loading shared-libraries.

3. The ACL shared-library is loaded if necessary and lisp_init is invoked. The operating system ensures that all shared-libraries that were linked into the ACL shared-library are loaded before lisp_init is allowed to start. If the ACL shared-library was linked in, lisp_init can be called immediately. Step 1 above assured that all proper links are done. See main.htm for further details.

4. lisp-init determines a heap (image) file, with extension typically .dxl. The heap file contains state information about the image, including whether a Pure Lisp Library will be used. The .dxl and .pll are loaded into memory in the following manner:

5. The C heap is mapped in, and C variables are set. If a .pll file is to be used this fact becomes known at this time. The C heap can not be relocated from where it was first built.

6. The .pll file, if present, is mapped in read-only. If it can't be mapped into the location it had been in a previous incarnation of Lisp, it is moved to another location. (Except when an image is built with build-lisp-image, there is always a previous incarnation of Lisp, perhaps the one that ran when the image was built).

7. The Lisp heap is mapped in. The best case is if the heap can be placed at the address and size that were specified by the build parameters. If those can't be satisfied, decreasing sizes are tried until either one is successful or the current commit level is passed (at the same address), followed by an operating-system-selected space of the built size, followed by decreasing sizes tried at OS selected locations until one is successful or the current commit level is passed (at which time Lisp startup fails). If the lisp-heap-size happens to be larger than the available swap, then only as much as can be actually allocated is used, as long as it is at least as large as the commit requirement.

8. Pointers are adjusted if the Lisp heap or the .pll file had to be moved. (The Lisp heap file might have to be moved because the Lisp heap start address is not available, e.g. The .pll file might have to be moved if the address in the previous Lisp invocation is unavailable.) All pointers to the expected locations of the Lisp heap file and the pll file are moved to the new locations. This step is performed all at once, to ensure proper pointer movement.

9. The Lisp starts. The process by which the Lisp starts is documented by the source file, <allegro directory>/src/aclstart.cl, which is provided in your distribution, and described in Section 12.0 of startup.htm. As one of the items in aclstart.cl, excl::reload-fix-entry-points (this function is not further documented) is called, which ensures that all system and user libraries are loaded if they are not already. This may involve performing a load on any libraries that were not loaded.

Now, there is a potential problem with the last step. If

• there are user libraries that are to be loaded, and

• the lisp-heap-size is specified as larger than the available swap space, and

• a contiguous address space is found for the Lisp heap that is also larger than the available swap,

then it is conceivable that there will be no swap left for the loading of the user libraries.

In this situation, it would be best for the executable to be programmed to pre-load the user librar(y/ies) so that the space is pre-allocated. (You do this by writing a customized main() that loads the needed libraries, see main.htm) But this presents the possibility that the user library might break up the contiguous address space for the Lisp heap, especially on Windows. The problem in intractable in general, but solvable in individual cases.

How can I tell where my image's heaps are located, and what size they are?

The ansi standard function room can be used for this. A description of the output generated by this function is described in section 2.4 Getting information on memory management using cl:room of gc.htm.

Knowledge of your heap locations is necessary to make use of the below suggestions for finding room for larger heaps or for relocating DLLs so they don't interfere with your heap growth. Likewise, if you are interested in increasing your heap size, you should have an idea of how much space you want based on your application requirements.

How can I tell what addresses are being used in my process space?

There is a C function in the lisp:

void memory_status_dump(char *file);

If you define a foreign call to this function and then call it while the lisp is running, it will give you a dump of every address in the process, and its status. A string argument will dump the input to the given file, whereas an argument of 0 will write to stdout.

For example, on Solaris the output would like look:

Mappings: 52, from struct prmap_t in sys/procfs.h:
0x00000000 - 0x0000ffff      65536  free
0x00010000 - 0x00017fff      32768  ---r-x offset = 0
0x00018000 - 0x00025fff      57344  free
0x00026000 - 0x00029fff      16384  ---rwx offset = 24576
0x0002a000 - 0x00033fff      40960  -b-rwx offset = 0
0x00034000 - 0x03ffffff   66895872  free
0x04000000 - 0x04705fff    7364608  ---rwx offset = 24576
0x04706000 - 0x04771fff     442368  ---rwx offset = 0
0x04772000 - 0x04899fff    1212416  ---rwx offset = 7389184
0x0489a000 - 0x04a49fff    1769472  ---rwx offset = 0
0x04a4a000 - 0x53ffffff 1331388416  free
0x54000000 - 0x54005fff      24576  ---rwx offset = 9093120
0x54006000 - 0xfe6fffff 2859442176  free
0xfe700000 - 0xfe7a7fff     688128  ---r-x offset = 0
0xfe7a8000 - 0xfe7b5fff      57344  free
0xfe7b6000 - 0xfe7bffff      40960  ---rwx offset = 679936
0xfe7c0000 - 0xfe7d7fff      98304  ---rwx offset = 0
0xfe7d8000 - 0xfe801fff     172032  free
0xfe802000 - 0xfe803fff       8192  ---rwx offset = 0
0xfe804000 - 0xfe903fff    1048576  free
0xfe904000 - 0xfe905fff       8192  ---rwx offset = 0
0xfe906000 - 0xfea05fff    1048576  free
0xfea06000 - 0xfea07fff       8192  ---rwx offset = 0
0xfea08000 - 0xfeb07fff    1048576  free
0xfeb08000 - 0xfeb09fff       8192  ---rwx offset = 0
0xfeb0a000 - 0xfec09fff    1048576  free
0xfec0a000 - 0xfec0bfff       8192  ---rwx offset = 0
0xfec0c000 - 0xfed0bfff    1048576  free
0xfed0c000 - 0xfed0dfff       8192  ---rwx offset = 0
0xfed0e000 - 0xfee0dfff    1048576  free
0xfee0e000 - 0xfee0ffff       8192  ---rwx offset = 0
0xfee10000 - 0xfef0bfff    1032192  free
0xfef0c000 - 0xfef0dfff       8192  ---rwx offset = 0
0xfef0e000 - 0xfef0ffff       8192  free
0xfef10000 - 0xfef11fff       8192  ---rwx offset = 0
0xfef12000 - 0xff00dfff    1032192  free
0xff00e000 - 0xff00ffff       8192  ---rwx offset = 0
0xff010000 - 0xff0bffff     720896  free
0xff0c0000 - 0xff0ddfff     122880  ---r-x offset = 0
0xff0de000 - 0xff0ebfff      57344  free
0xff0ec000 - 0xff0effff      16384  ---rwx offset = 114688
0xff0f0000 - 0xff0f9fff      40960  ---rwx offset = 0
0xff0fa000 - 0xff0fffff      24576  free
0xff100000 - 0xff1a3fff     671744  ---r-x offset = 0
0xff1a4000 - 0xff1b1fff      57344  free
0xff1b2000 - 0xff1b9fff      32768  ---rwx offset = 663552
0xff1ba000 - 0xff1bbfff       8192  ---rwx offset = 0
0xff1bc000 - 0xff1e3fff     163840  free
0xff1e4000 - 0xff1e5fff       8192  ---rwx offset = 0
0xff1e6000 - 0xff1effff      40960  free
0xff1f0000 - 0xff1f3fff      16384  ---r-x offset = 0
0xff1f4000 - 0xff1fffff      49152  free
0xff200000 - 0xff27ffff     524288  ---r-x offset = 0
0xff280000 - 0xff28dfff      57344  free
0xff28e000 - 0xff297fff      40960  ---rwx offset = 516096
0xff298000 - 0xff29ffff      32768  ---rwx offset = 0
0xff2a0000 - 0xff2affff      65536  free
0xff2b0000 - 0xff2b1fff       8192  ---rwx offset = 0
0xff2b2000 - 0xff2bffff      57344  free
0xff2c0000 - 0xff2c3fff      16384  ---r-x offset = 0
0xff2c4000 - 0xff2d1fff      57344  free
0xff2d2000 - 0xff2d3fff       8192  ---rwx offset = 8192
0xff2d4000 - 0xff2dffff      49152  free
0xff2e0000 - 0xff2e7fff      32768  ---r-x offset = 0
0xff2e8000 - 0xff2f5fff      57344  free
0xff2f6000 - 0xff2f7fff       8192  ---rwx offset = 24576
0xff2f8000 - 0xff2fffff      32768  free
0xff300000 - 0xff307fff      32768  ---r-x offset = 0
0xff308000 - 0xff315fff      57344  free
0xff316000 - 0xff319fff      16384  ---rwx offset = 24576
0xff31a000 - 0xff31ffff      24576  free
0xff320000 - 0xff321fff       8192  --srwx offset = 0
0xff322000 - 0xff32ffff      57344  free
0xff330000 - 0xff345fff      90112  ---r-x offset = 0
0xff346000 - 0xff353fff      57344  free
0xff354000 - 0xff355fff       8192  ---rwx offset = 81920
0xff356000 - 0xff35ffff      40960  free
0xff360000 - 0xff365fff      24576  ---r-x offset = 0
0xff366000 - 0xff373fff      57344  free
0xff374000 - 0xff375fff       8192  ---rwx offset = 16384
0xff376000 - 0xff37ffff      40960  free
0xff380000 - 0xff387fff      32768  ---r-x offset = 0
0xff388000 - 0xff395fff      57344  free
0xff396000 - 0xff397fff       8192  ---rwx offset = 24576
0xff398000 - 0xff39ffff      32768  free
0xff3a0000 - 0xff3a1fff       8192  ---r-x offset = 0
0xff3a2000 - 0xff3affff      57344  free
0xff3b0000 - 0xff3cdfff     122880  ---r-x offset = 0
0xff3ce000 - 0xff3dbfff      57344  free
0xff3dc000 - 0xff3ddfff       8192  ---rwx offset = 114688
0xff3de000 - 0xff3dffff       8192  ---rwx offset = 0
0xff3e0000 - 0xff443fff     409600  free
0xff444000 - 0xff445fff       8192  ---rwx offset = 0
0xff446000 - 0xffbe5fff    7995392  free
0xffbe6000 - 0xffbeffff      40960  s--rwx offset = 4294942720
    

Note that each section of memory is grouped by how it is currently mapped; the four characters at the end are flags, with a dash representing not available, and one of r, w, x, and c for read, write, execute and copy-on-write (shared until written, when it will get a private copy).

The problem with memory_status_dump is that it doesn't always identify what object is being mapped into any particular address area (such information depends on what's available on a per platform basis). It seems reasonably intuitive what an "object" is by all of the maps on all architectures; each address range constitutes one mapping. However, it is impossible to tell what the objects actually are, and sometimes when two address ranges are exactly adjacent to each other, the higher address range is likely an extension of the lower address range. This is important in deciding what objects to try to identify when deciding how to move things around. For example, if the C heap starts at 0x54000000 and the map has two ranges next to each other, one from 0x54000000 to 0x54007fff and one from 0x54008000 (i.e 0x54007fff + 1) to 0x5401dfff, then it is likely that the C heap was extended once, and thus that if the C heap is moved it will take care of both address ranges.

For x86 windows, we recommend the use of Process Explorer (previously hosted by www.sysinternals.com, but now distributed by Microsoft). This utility provides a lot of information and has a great user interface.

It is available for download here.

Download and install this program. Then, start Allegro CL and run Process Explorer. This will bring up a dual-paned window, the top half of which will show a list of running processes. Click on `allegro.exe' (or `mlisp.exe' or `alisp.exe'), and in the lower pane you will be shown what objects have been allocated in the processes address space.

Some Process Explorer configuration tips:

• make sure that there is a lower pane showing (menu: View -> Show Lower Pane has check mark)

• make sure that the lower pane shows DLL info (menu: View -> Lower Pane View -> DLL has check mark)

• in the lower pane, right click on the column headers and pick SelectColumns. Check at least Name, Path, Base address, Mapped size, Mapping type

• make sure that mlisp.exe is selected in the upper pane

• make sure that the lower pane is sorted by ascending Base address

• then, finally, save to a file.

For other platforms, the situation is more complicated, but there are usually operating system tools for interpreting the data.

[Windows only] How do I move DLL in memory so that it doesn't conflict with the Lisp heap?

The preferred Lisp heap starting address is 0x20000000 on x86 Windows (the heap grows to higher memory addresses). If, using the Process Explorer program (discussed above, an x86 Windows application), you find that the Lisp heap cannot grow to the size that you desire because of a DLL that is in the way, you can use the `editbin' program from Microsoft Visual C++ to move the default base address of the DLL. For example, if foo.dll is in a bad location, according to Process Explorer, and you found foo.dll to live in the c:\winnt\system32 directory, then at a DOS prompt type (assuming 0x65000000 is the beginning of a free address range large enough to accomodate the DLL):

cd c:\winnt\system32
editbin foo.dll /rebase:base=0x65000000
    

and the next time Lisp starts up, Windows will try to locate foo.dll at 0x65000000 instead of the previous inconvenient address that interfered with the Lisp heap. Note that you may need to disable related programs or boot into safe mode in order to perform this operation. It is generally not possible to modify such files when they are active and loaded into a process address space.

Note that the base address for a DLL is only advisory; if the space is already used, Windows will locate it to some other location. By right-clicking on a DLL and selecting "Quick View" (if Quick View support is installed), you can see (among other things) the base address currently assigned to the DLL.

What does the "Temporarily scaling back lisp reserved region from XXX to YYY bytes." mean?

It means that some other program has grabbed part of the address space that Lisp intended to use. This message is much more common on Windows, where various programs can grab predefined address ranges in all running programs. The Iomega Zip/Jaz tools and PGP are two programs known to do this. There are four things that you can do to handle this situation:

1. Ignore the warning. This means that your lisp heap might be significantly smaller than intended. You can tell just how big by dividing YYY above by 1048576, which will give you the size of the heap in MB.
2. Uninstall the application. Sometimes the application isn't really needed. On Windows, the Iomega tools on a SCSI system, for example, are not really needed--Windows has native support for these SCSI devices.
3. [Windows only] "Rebase" the DLL to use another address. See How do I move DLL in memory so that it doesn't conflict with the Lisp heap? for more information. Look around address 0x65000000 for a free slot to which you can rebase the DLL. If addresses in that range are taken, find the Windows DLLs (e.g., COMCTL32.dll) and pick a location around them for the DLL to live.
4. Rebuild your ACL images using a different range of addresses for the Lisp heap. On UNIX, see How can I tell what addresses are being used in my process space? for more information.

The arguments to build-lisp-image that you will need to specify are :lisp-heap-start, :lisp-heap-size, :c-heap-size and :c-heap-start.

What should I know when choosing non-default heap locations?

Both the Lisp and C heap must remain in one individual contiguous piece. (i.e a single contiguous lisp heap, and a separate single, contiguous C heap). You need to find a large enough gap in the address space to cover all growth expectations of your application. Most 'large' apps need lots of room for the lisp heap to grow and very little for the C heap. Your mileage may vary.

On Unix platforms, the OS allocates memory in the sbrk region, which typically resides directly after the mapped location of the lisp executable. If you restrict this region too much, you run the risk of causing your application to fail with an "unable to allocate memory" error that will usually crash the lisp. This malloc region is used by low level OS system calls and typically by any foreign libraries included in your application.

Combined with the knowledge of your current heap locations and sizes, you can use memory-status-dump (see earlier question) to find free regions in the process address space.

How do I build an image with non-default heap sizes and/or locations?

The typical way to build new images is via the function build-lisp-image. It accepts the following four keyword arguments that are used to relocate the lisp and c heaps from their default locations and sizes:

• :lisp-heap-start

• :lisp-heap-size

• :c-heap-start

• :c-heap-size

The following two keyword arguments let you further shape the lisp heap once it has been allocated.

• :oldspace

• :newspace

Once you have chosen values for these parameters (see previous questions in this section for help doing so), simply include them in your call to build-lisp-image. When starting the new image, a call to room should show that the lisp heaps have been relocated.

How do I build default images provided by Franz with non-default heap sizes and/or locations?

A common problem with developers of 'large' applications is that default development images such as alisp.dxl and allegro.dxl are built with default heap locations. Further, even if developers customize the heap locations and sizes of these apps, their changes are undone when they update patches and run update.sh or update.exe.

To this end, we have added the following environment variables that build-lisp-image will check for when creating a new image.

• ACL_BUILD_LISP_HEAP_START: :lisp-heap-start argument

• ACL_BUILD_LISP_HEAP_SIZE: :lisp-heap-size argument

• ACL_BUILD_C_HEAP_START: :c-heap-start argument

• ACL_BUILD_C_HEAP_SIZE: :c-heap-size argument

• ACL_BUILD_NEWSPACE: :newspace argument

• ACL_BUILD_OLDSPACE: :oldspace argument

Arguments values should use lisp hex notation (#x).

It is recommended that users not modify their general development environment when making use of these environment variables as that can result in undue confusion and unexpected results. Instead, it is recommended that a new script (or batch) file be created with a name not used by Allegro CL, such as 'my-update.sh' or 'my-update.bat', which invokes update.sh or update.exe depending on the platform. This script file can then set these environment variables as desired before invoking the Franz provided update script.

The user then need only remember to run their personalized update script and their heap settings should persist even when patches are updated.

Can I specify heap locations and/or sizes when starting lisp?

No. The C heap in particular is not relocatable, so this approach is not feasible. The Lisp heap is relocatable--and occasionally will shrink if the reserve space is not available--but due to the C heap restriction, the only option would be to lower it in memory. This being only half a possible solution, we have not opened up this limited functionality.

Garbage Collection

My memory gobbling loop causes the gc to perform badly. Why?

The loop you are running is likely not releasing any bytes of heap for the amount it is allocating. A loop similar to

(compile
 (defun gobble ()
   (loop for x from 1 to 1000000000000000000000000
       with k = 0
       do
	 (incf k)
	 (when (= k 10000000)
           (print ".")
           (setq k 0))
       collect x)))
(gobble)
    

is likely generating almost no garbage. Allegro CL's garbage collector makes decisions about how much space to allocate based on the gsgc-parameters that are currently set, and the effect of this loop is to throw the calculations into a non-linear area where the only thing it can do is to cause the heap to grow at a huge rate of increase.

Most real programs never run this way; either they don't cons at all, or at least some of the consing done is ephemeral, even if only a few percent of the total consing. And, of course, any program which continues to cons permanent space should be considered leaky, unless it is explicitly a test to gobble up all of memory, and to see how such gobbling progresses.

To get better gobbling behavior, allocate at least some of the space in such a way that it becomes garbage. This allows the gc to operate in a more natural manner, and will cause a smoother transition toward the inevitable heap-exhaustion:

(compile
 (defun gobble2 ()
   (loop for x from 1 to 1000000000000000000000000
       with k = 0
       do
	 (incf k)
	 (when (= k 10000000)
           (print ".")
           (setq k 0))
         (when (zerop (mod k 10))
           (cons x nil)) ; waste a cons
       collect x)))
(gobble2)
    

Finally: the reason why heap grows so quickly is because newspace must hold all of the objects that have been allocated until they are to be tenured, even though it is known that none of them will die, and so the newspace must continually grow to hold the new allocations as well as the objects waiting to be tenured. An explicit gc with tenuring periodically will serve to flush the newspace out, and will also keep it to a reasonable size, thus making the march toward heap-exhaustion even more smooth:

(compile
 (defun gobble3 ()
   (loop for x from 1 to 1000000000000000000000000
       with k = 0
       do
	 (incf k)
	 (when (= k 10000000)
           (print ".")
           (setq k 0)
           (gc :tenure)) ; clear newspace
         (when (zerop (mod k 10))
           (cons x nil)) ; waste a cons
       collect x)))
(gobble3)
    

Foreign Functions Interface

How do I pass and return 64-bit integers through the FFI?

Allegro CL does not support 64-bit integers in 32-bit lisps, since we try to be compatible on all architectures on which we run, and since there are so many different extensions to C to allow for 64-bit integers, which are of course longer than a "long" type.

You can work around this by defining a struct containing two 32-bit values

(def-foreign-type unsignedlonglong
     (:struct (high :unsigned-int) (low :unsigned-int)))

(defun unsignedlonglong-value (x)
  (+ (ash (fslot-value-typed 'unsignedlonglong nil x 'high) 32)
     (fslot-value-typed 'unsignedlonglong nil x 'low)))
    

When you define your foreign function, have it return the struct, rather than the integer, and have the lisp reconstruct the number using the function above. You can then have the lisp reconstruct the number using the function above.

For passing 64-bit values, you can write a routine to pack 64-bit values into the struct. The struct returned should be passed to the foreign function call.

(defun value-unsignedlonglong (x)
  (let ((inst (ff:allocate-fobject 'unsignedlonglong)))
    (setf (ff:fslot-value-typed 'unsignedlonglong nil inst 'high)
	  (ash x -32)
	  (ff:fslot-value-typed 'unsignedlonglong nil inst 'low)
		 
	  (logand x #xffffffff))
    inst))
    

The above code could be further generalized to allow for multiple allocation types. Users are cautioned to be careful not to introduce memory leaks into their application.

CLIM

How can I replace the lesstif installed with RedHat Linux 7.2 with openmotif (required for CLIM)?

This question is answered here, in the Linux (architecture specific) section.

A (require :clixm) fails because it can't find libXm.so.3, but I have libXm.so.4 installed. What do I need to do?

We have had reports of this occurring on Fedora Core 5 systems. All of our software is built on the oldest OS that we support for a given platform. When new systems come out post-release, this occasionally occurs.

The solution is to create a symbolic link from libXm.so.3 to libXm.so.4. For example:

% ln -s /usr/X11R6/lib/libXm.so.4 /usr/X11R6/lib/libXm.so.3
    

Once this link is created, (require :climxm) should properly locate the required Motif library and complete successfully.

Composer

When starting Composer I get the error 'Error: "Connection refused" (errno 111) occurred while creating a local socket and connecting to a remote host ... on port 6000.'

Recent Linux distributions deliver X servers that no longer listen for tcp/ip connections, even on the local host. Composer accesses X displays via the CLX (Common Lisp X) module, which until recently only supported X communication via tcp/ip.

A patch is now available for Allegro CL 8.0 that adds support for unix domain socket connections to the XCW and CLX modules. The LOG entries for these patches can be read at:

• code/clx.001

• update/pdh001.001

These patches will be automatically downloaded if you run sys:update-allegro

An alternative fix is to re-enable tcp/ip connections in your X server. In SuSE Linux 9.2, for example, set the environment variable DISPLAYMANAGER_XSERVER_TCP_PORT_6000_OPEN in /etc/sysconfig/displaymanager to "yes" to turn tcp/ip connections back on the next time X is started.

This fix will propagate to future releases of Allegro CL, where a patch will no longer be needed.

Compatibility between 32 and 64-bit versions of Allegro CL

What changes are needed to move from a 32-bit to 64-bit Allegro CL?

Very little.

Why does my 64-bit foreign call cause a SIGSEGV?

If your code works fine on 32-bit versions of Allegro CL, then it is most likely because of the difference in size between :int and :long.

The :returning keyword argument to def-foreign-call is a key piece of the solution, where the return value has been declared as an :int rather than a :long or an :unsigned-long. Passing an :int into a foreign call is also dangerous, where the expected types might mismatch.

The real problem is that C defines the default declaration to be int (i.e. if a variable or function return is not declared, it is assumed to be int). But in a 64-bit application, the equivalent integer (an integer whose size matches that of a pointer) is long not an int, and so there is a tendency to underdeclare C functions.

And because our def-foreign-call interface tries to match the interface for C, the same issue occurs - but because in our def-foreign-call, where there is no way to check against the C code like a C compiler can (and can thus issue warnings about possible data loss when assigning a pointer to an int variable) the defaults become a disadvantage.

Misc

Sometimes TIME results produce negative values. Why?

As an example, see the below output

CL-USER(12): (time (fib 30))
; cpu time (non-gc) 36,784 msec user, 140 msec system
; cpu time (gc)     3,994 msec user, 10 msec system
; cpu time (total)  40,778 msec user, 150 msec system
; real time  51,274 msec
; space allocation:
;  60,582,248 cons cells, -934,589,280 other bytes, 0 static bytes
1346269
CL-USER(13):
    

The problem is with the overflow of machine integers. On 32-bit platforms, the allocator, which resides in low level C code, keeps track of counts using 32-bit variables. As these allocations occur over time, these variables can in fact overflow resulting in the negative output seen above. It would be a simple matter to adjust such values in the case where an overflow occurs only once (the typical case), by adding #x100000000 (4 GB) to the negative value. But because we cannot determine how many overflows actually occured, we choose not to adjust in all cases where negative space is reported as that would give wholly false (but apparently valid) results in some cases. We recommend trying the adjustment (adding #x100000000) and seeing if the result looks reasonable.

Multiprocessing

In the context of Common Lisp, what is Symmetric-multiprocessing?

An SMP Lisp is one in which performance can scale with processor count as well as with processor speed. It makes "just add more cpus" an option for dealing with increased load. It's pretty well known that a problem has to factor in an unusual way for this to work, whatever language and library is used. Since support functions (file access, paging, network communications, etc) are already using extra processors, every application that uses these functions is getting some benefit from any extra CPU's on the system.

The iconic application for SMP is transaction processing. Network speed and current cycle rates (factored by cpu count) can feed a huge number of transactions to an application. Allowing parallel execution of several transactions could boost transaction throughput, as long as a serializing resource (data base access for example) doesn't become the real bottleneck.

Can Allegro CL run more than one native thread at a time on machines with more than one processor?

On Windows, where Allegro CL processes are implemented with native operating system threads, a single Lisp thread and any number of foreign threads independent of the Lisp heap may run at the same time on different processors. [A "foreign thread" is an ACL process that has made a foreign call and released the heap or an independent thread started by a call to a foreign library function. See this for more information.]

Currently, on UNIX, Linux and Mac OS X, Allegro CL processes are not implemented with native operating system threads.

We are working on a Symmetric-multiprocessor implementation of Allegro CL, but the release date is currently unknown. See below for a discussion of the challenges of implementing an SMP Lisp.

To take advantage of multiple processors you have these choices:

• Using fork() to simulate SMP.

• Multiple Lisp processes communicating via ACL RPC.

• Multiple Lisp processes communicating via sockets. That is, the same as the ACL RPC solution, but using your own socket protocol. If your needs are simple and performance is an issue, this might be the best solution.

In the near future we plan to add at least one solution to the above list. Watch this space.

Why is it taking so long for you to implement SMP in Allegro CL?

The implementation of a full SMP Lisp requires two parts:

1. Mapping from Lisp threads to native operating system thread library. On Linux this means `NPTL'. On Windows this is the WIN32 library.

2. Making the compiler and library of Lisp code written for the implementation SMP "thread safe". By "thread safe", we mean the orderly updating of global resources. For example, two threads adding to a hash table, changing a symbol's value or defining a method on a generic function. These operations must be done properly so the Lisp program doesn't unexpectedly crash.

We cannot call Allegro CL "SMP capable" until both of the above are completely finished. It would be unacceptable to us to do #1 and release a new public version of Allegro CL, knowing full well that because of #2 customer applications would randomly crash from time to time. This is completely unacceptable for an industrial strength Common Lisp.

In addition to thread safety, a true SMP Lisp will necessarily slow down the scalar speed of the implementation. That is, thread safety incurs a significant performance penalty compared to a non-SMP system. To insure thread safety, locks must be added, although with careful engineering effort many such globally-visible operations can be done with lock-less synchronizing instructions. Still, even these instructions are slightly slower than the simple store instructions, and will represent slower scalar operation.

Even when we offer a full SMP Lisp, the speed trade-off between the SMP and non-SMP versions will be a real issue for some. The design-level methods for SMP-like behavior discussed above are sometimes preferrable to the actual SMP overhead. Because of this, we expect the need to maintain the current versions of Allegro CL (the non-SMP ready ones) for quite some time, and offer along side that version one that is SMP capable. (At the source level we will not have two different versions--our single source will be compiled into two different products, one SMP and the other not.)

We are actively working on a full SMP Lisp, one that combines the benefits of SMP with as few locks as possible, while yet maintaining a user programming interface that requires as few special programming situations as possible.

Lastly, we often hear "SBCL has done it, so why can't you?" To quote the SBCL documentation:

Large amounts of the SBCL library have not been inspected for thread-safety. Some of the obviously unsafe areas have large locks around them, so compilation and fasl loading, for example, cannot be parallelized. Work is ongoing in this area.

What SBCL has done is #1 above with a little of #2. As of December 2007 they have big locks around the compiler, fasl loader, and foreign loader. They must single-thread those because they cannot otherwise ensure their use won't cause crashes or (even worse) less obvious failures. As we stated above, we do not believe this is a good strategy for Allegro CL. We will release an SMP Lisp when it works completely. Also, we believe that when they finish making their libraries thread-safe, they will find significant performance issues that need to be solved.

Since you use native threads on Windows, why isn't that an SMP Lisp?

Because the locking of the heap is all or nothing. That is, within a single Lisp the heap cannot be shared between two Lisp processes running at the same time. An SMP Lisp requires finer grained locking when modifying globally accessible objects, and that work has not yet been completed.

For more information see this.

Windows (architecture specific)

Does Allegro CL work on Windows Vista?

Yes, but on Vista there is a security model which disallows installation of files into c:\Program Files\ if you do not have Administrator priviledges, which is where Allegro CL is installed by default. This may affect your experience using Allegro CL in the following two circumstances:

1. If you have a habit of working in the Allegro directory (in C:\Program Files\) you will get get errors every time you try to modify a file. You will need to find another location for your working files or run Allegro CL under elevated priviledges. To do this, navigate to the Allegro CL start menu and right-click on the image you'd like to start. Select Run as administrator. When you select Run as administrator you will need to answer in the affirmative to the User Account Control dialog boxes Vista pops up. This will grant you the permissions needed to modify or create files in this directory.

2. When installing Allegro CL patches. A description of how to update your distribution can be found by following this link.

My lisp immediately crashes a few seconds after startup. What's causing this?

Current versions of Microsoft operating systems, such as Vista, XP, and Server 2003 come with a security feature enabled called Data Execution Prevention. While it helps protect against a common class of malicious attacks, it also prevents our lisp from running! To correct this, firat verify that DEP is enabled and is what is causing the problem.

• Right-click on 'My Computer' and choose Properties.

• In the dialog that appears, select the 'Advanced' tab, and then in the Performance section of the dialog, press the 'Settings button.

• Another dialog will appear. Choose the 'Data Execution Prevention' tab.

There are a couple of options here.

• Turn on DEP for essential Windows programs and services only.

• Turn on DEP for all programs and services except those I select.

If DEP is only enabled for essential Windows programs, then DEP is not causing the problem. Please see the FAQ entry on How to report Bugs and submit a report to support@franz.com. If DEP is enabled for all programs and there are no lisp exceptions, it is likely the cause.

Choosing the first option is the simplest, since Allegro CL provides a number of different executables from which Lisp can be used. Also, the update process creates a temporary executable called update.exe when updating images to contain new patches. If you decide to add DEP exceptions for Allegro CL, you will need to create a temporary file named update.exe in the Allegro directory and add it using the DEP dialog.

Why can't I use `dir' with run-shell-command?

On Windows, excl:run-shell-command executes programs but does not invoke shell commands. The function is therefore misnamed for Windows. It is called run-shell-command to provide cross-platform compatibility between Windows and Unix and back compatibility for ACL4.3.x users. It does behave differently on Windows and Unix. The difference is related to differences between UNIX and Windows. See the documentation page for excl:run-shell-command for details on this point.

How do I control the stack size on Windows?

The default stack reserve size for the lisp process is 16MB.

The stack-size process attribute was supposed to control the stack allocation for the process. What the Windows documentation did not make clear, unfortunately, was that the running process can control the amount of swap space initially committed to the stack, but cannot select the amount of address space reserved for the stack.

The address space to be reserved for each thread's stack is a property of the executable image being used. The property is specified at link time, but it can be altered without relinking, through the EDITBIN program that is part of the VC+ development package.

To make each thread allocate a 4Meg stack, the command would be (for the executable allergo.exe, repeat the command for other executables):

editbin /stack:4000000 allegro.exe
    

Unfortunately, MS Windows offers no way to have different reserve sizes for different threads in the same process, so all the threads have to be big enough to support the deepest computation on any thread.

How do I get ANSI ACL (rather than Modern ACL) to start when I double-click on an lpr file?

The Windows registry associates file types with programs, so that double clicking on a file of a particular type initiates a program in a clearly defined way. (Thus, typically double clicking on a file with type txt typically initiates the Notepad utility in Windows, opening the file clicked on.) As defined by the installation procedure, lpr files (the files defining projects) are associated with allegro.exe/allegro.dxl, the Modern version of Allegro CL with the IDE. Double-clicking on an lpr file thus starts the Modern image with the IDE, and opens the project associated with the lpr file.

If you want the ANSI Allegro CL with the IDE (allegro-ansi.exe/allegro-ansi.dxl) to start instead, you must modify the registry.

The simplest way to do this is with a Windows tool. While Windows menu choices move about, try opening the View menu on the Windows Explorer and click on the Options... item or try the Tools menu and the Folder Options choice. This will display an Options dialog. Click on the File Types tab, and find Lisp project file in the displayed list of types. Select it and click on the Edit... button. Select open in the Actions pane and click on the Edit... button. Modify the text in the Application used to perform action field, changing allegro.exe to allegro-ansi.exe and allegro.dxl to allegro-ansi.dxl. Click on the OK buttons on the open dialogs to complete the change. On Windows 2000, open the Tools menu and click on the Folder Options item, and choose the LPR extension, click on the Advanced button, click on the open action, click on the Edit... button, find the associated command line, and modify it.

Why is the compiler complaining about a missing in-package form when I am certain that my offline file starts with one?

When working with Windows offline files, the following situation has been observed. As a result, we recommend that in general, one not develop using offline files.

1. A file, foo.cl, is compiled.

2. The compiler signals an error that goes to a break loop.

3. Go to an Emacs buffer where the file foo.cl is showing.

4. Make some correction to the file and save it.

5. :continue from the break.

6. The compiler claims that there is no in-package.

7. The file foo.cl on disk appears to be all hex zero bytes.

If the file foo.cl is still in the Emacs buffer, it can be saved again to recover the content on disk; but if the Emacs buffer was killed, the content of the file is lost.

There seems to be no way for ACL to detect this situation and thus avoid it or warn the user about the possible danger. The only solution is to use caution when working with offline files.

We have found that using an alternate tool such as WinSCP can be one way to avoid this problem while retaining the ability to synchronize remote files conveniently.

Linux (architecture specific)

On Allegro CL startup I get the error "cannot restore segment prot after reloc: Permission denied". Why?

NOTE: sometimes the error can be "segmentation fault".

This error occurs when SELinux (Security-Enhanced Linux) is enabled. This is a security extension originally developed by the NSA that comes enabled by default in some newer linux distributions. One of the security features changes the way shared libraries are loaded by default, and this is preventing Allegro CL from properly starting.

The best solution is to run the following command to enable Allegro CL to load its shared library:

# chcon -t textrel_shlib_t /usr/local/acl81_express/libacli817t.so
    

You need to substitute /usr/local/acl81_express with the actual path to your Allegro CL installation directory. Also, libacli817t.so is the name of the Allegro CL Express Edition shared library, and for other versions of Allegro CL the name might be different. Run the above command on any file named .so in the Allegro CL directory.

Another solution is to disable SELinux entirely. This is done by either of these methods:

• As root, run system-config-securitylevel, go to the SELinux tab, and change the SELinux setting to Disabled.

• Set the line SELINUX=disabled in /etc/sysconfig/selinux.

On which x86 (i.e., Intel Pentium and friends) Linux versions do the currently supported versions of Allegro CL run?

Allegro CL runs on a large number of kernels, for which we do not provide a comprehensive listing. See our Platform Listing for current information about kernel or glibc restrictions.

How can I replace the lesstif installed with RedHat Linux 7.2 with openmotif (required for CLIM)?

Allegro CL 8.x CLIM requires Motif 2.1, a free version of which is available from www.openmotif.org.

Although the default installation of RedHat Linux 7.2 includes installation of lesstif (a variant of Motif that does not work with CLIM), there is an installation mode where you can alter the default package installation list and therefore prevent the installation of lesstif. If you do this, you can simply install openmotif Motif 2.1 after obtaining it from www.openmotif.org.

However, once lesstif is installed, you should remove it before installing openmotif Motif 2.1. To uninstall lesstif, enter

% rpm --erase lesstif
    

The rpm program will notify you if removing the lesstif package would break anything (and it will not continue with removing lesstif). rpm will indicate which packages depend on lesstif. Please contact Franz Inc. (sending mail to support@franz.com) for assistance if there are problems removing lesstif.

Once lesstif is removed, openmotif Motif 2.1 can be installed following instructions received with the software.

Mac OS X (architecture specific)

The downloaded .dmg.bz2 file appears to be corrupt. Why?

It is a known bug in Safari on Mac OS X that it sometimes adds file extensions to downloaded files. In this case, it is adding a .bz2 file extension when there should not be one.

You can fix the problem by renaming the .dmg.bz2 file to have extension .dmg.

Why do I get crash reports when running 32-bit PPC Allegro CL?

This is a known bug in Mac OS X. Programs which catch Unix-style signals and handle them will still result in "crash" reports being generated erroneously by the CrashReporter.

See http://developer.apple.com/technotes/tn2004/tn2123.html#SECLIMITATIONS and note close to the end of the page under the heading "CrashReporter Limitations", specifi