Urania

One of the most important programs that hadn’t been ported to Leopard was AppleJack. AppleJack is an awesome program written by Kristofer Widholm for system maintenance that you can run in Single User Mode. In a nutshell, it makes it easy to repair disk permissions, clean out foobared cache files, bad preference files, etc… all automatically. Its easy to install and to run it,

1. Reboot your Mac and hold down Cmd-S during the boot until the text booting dialog appears.
2. Run applejack. I enter applejack auto restart to automatically do all the system maintenance then restart into normal multi-user (graphics interface) mode.

AppleJack has saved my bacon more than once. I use it on those rare occasions where I have to do a forced shutdown of my Mac, but it has also fixed problems due to bad preferences or messed up font caches for me. I just tried out this version under Leopard on my MacBook Pro and it worked like a charm, about 5 minutes start to finish and it found a variety of permissions problems and fixed them. Best part is its free, so you have no excuse for not using it.

While the main site hasn’t been updated to reflect the presence of the new files, you can download AppleJack for Tiger or Leopard (version 1.5) here. There are also versions for Jaguar and Panther. Basically, its a must have tool for almost any MacOS version!

Scisoft OSX Intel 2008.8.1 was released about a week ago. I have been working with Nor Pirzkal for the last few months beta-testing this version and trying to make sure my concerns about the previous version were addressed. Nor’s blog post about the update states

There are a few changes in this version and all packages have been updated to the latest available versions. Starting with this version, Scisoft is installed in /Applications/scisoft (it previously was installed in /scisoft). Make sure that you remove any old /scisoft installation and properly update your startup files to source the Setup.csh or Setup.bash from their new locations.

In the README file you are told

Once the collection is successfully installed csh and tcsh users should invoke the command:

source /Applications/scisoft/all/bin/Setup.csh

to gain access to all the software and configure their environment correctly.

Users of the “bash” shell should instead use:

. /Applications/scisoft/all/bin/Setup.bash

Personally, to avoid problems in tcsh with a script breaking or with modifying my PATH multiple times by repeated execution of the Setup.csh script, I use the following line in my .tcshrc to first check for the existence of the Setup.csh script and SCISOFT environmental variables before executing it.

if (! $?SCISOFT) then
test -r /Applications/scisoft/all/bin/Setup.csh && source /Applications/scisoft/all/bin/Setup.csh
endif

Permission Problems Persist
This version addresses all the issues I noticed with Scisoft OSX 2008.5.1 except for one annoying one, the reassignment of ownership of the enclosing directory on installation. If you installed the old version of Scisoft OSX, it would reassign ownership of the root (/) directory to the second user on the system (in my case, since their is none, it showed the user as “502″). I beta-tested various versions of Scisoft OSX but Nor was not able to stamp out this particular problem. I believe Nor has come to the conclusion after extensive testing that this is a problem with Apple’s software for constructing installer packages. By moving the Scisoft OSX installation to /Applications/scisoft, it is the /Applications directory that gets its ownership changed instead of the root directory, which is less damaging. However, I would strongly suggest checking the ownership of the /Applications directory afterward and if isn’t owned by an administrative user, set it as such using:

sudo chown username:admin /Applications

(where “username” is the primary administrative users username, in most cases, your username) to perform the repair.

A Quick-Fix for any Legacy Scisoft-related scripts
If you had a few scripts that relied on Scisoft OSX being located in /scisoft and you don’t want to edit them all is you can make /scisoft point to /Applications/scisoft. This can be accomplished by first moving the old version of Scisoft OSX before installing 2008.8.1 via the command line:

sudo mv /scisoft /scisoft_old

and then once you have installed the new Scisoft OSX in 2008.8.1, create a symbolic link from the old location to the new by typing (again from the command line)

sudo ln -s /Applications/scisoft /scisoft

This will allow any scripts that refer to items in /scisoft to continue to work for the most part.

What’s New?
An investigation of the /Application/scisoft/i386/Packages directory as well as the NEWS file reveals the following changes to this version of SciSoft OSX over the 2008.5.1 version.

• DS9 updated from 5.1 to 5.3beta
  
• FV updated from 5.1 to 5.2.1
• ATLAS updated to version 3.8.2
• MIDAS updated to 08FEBpl1.1
• cdsclient was updated to version 2.87
• OpenMotif updated to 2.1.32_compat
• cfitsio library updaed from 3.090 to 3.090 (the current version is 3.100)
• pango library updated from version 1.20.2 to 1.21.3
• pixman library updated from 0.10.0 to 0.11.2
  
• expat library updated from 2.0.0 to 2.0.1
  
• fontconfig library updated from 2.3.2 to 2.6.0
• freetype library updated from 2.2.1 to 2.3.6
• gettext library updated from 0.14.5 to 0.17
• gtk+ updated from 2.12.9 to 2.12.10
• libpng library updated from 1.2.10 to 1.2.29
• netCDF library version 3.6.2 added
• pkg-config updated from 0.20 to 0.23
• TclTk package updated from 8.4.13 to 8.4.19
• wcstools library updated from 3.6.4 to 3.7.3 (current version is 3.7.5)
• IRAF package rvsao updated t version 2.5.7 to 2.6.4
• Python was updated to 2.5.2 and the following Python libraries were updated:
  
• • ScientificPython updated to 2.6.1

Minor Glitches
The only other minor glitch I noticed was that /Applications/scisoft/i386/bin doesn’t appear to have a ds9 binary installed. You can fix this by installing the X11 version of ds9 there or by linking to the Aqua version of ds9 that was installed using the command line

ln -s "/Applications/scisoft/i386/Applications/SAOImage DS9.app/Contents/MacOS/ds9" /Applications/scisoft/i386/bin/ds9

What do you know, one reboot later and MacPorts Apache2 server with MacPorts PHP is throwing the same errors as the built-in PHP server. When I try to create a graphic using typography, I get

The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.

It wasn’t complaining before the reboot. Maybe Apple’s security kicked in after the reboot although for the life of me I can’t understand why. I’ve continued investigating and in Apple’s Discussion boards I found there was a string of comments about this problem. One user, Bill Eccles, seemed to isolate the problem and his description matches the symptoms I am seeing:

Finally, there’s a big problem with FreeType. As I discovered, anytime FreeType fonts are used by GD, they apparently make a Carbon API call of some sort. Problem is, Apache2 uses fork() without a corresponding exec() and, upon calling PHP/GD/FreeType, the Carbon call in FreeType causes Apache2 to crash. It shows up in the error_log as

The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
Mon Nov 26 12:38:11 2007 notice child pid 304 exit signal Trace/BPT trap (5)

and in the system.log as

Nov 26 13:12:00 shr-g5 ReportCrash664: Formulating crash report for process httpd659
Nov 26 13:12:02 shr-g5 ReportCrash664: Saved crashreport to /Library/Logs/CrashReporter/httpd_2007-11-26-131145_shr-g5.crash using uid: 0 gid: 0, euid: 0 egid: 0

I didn’t discover the cause–I can’t take credit for this one.

At this point he provides a link to a blog that was just reposting mailing list digests and as such got pulled down. I think I have found the original thread about the problem on the FreeType mailing lists here. Bill then suggests the following solution (this is not complete):

[I]nstall FreeType 2 without the Mac-specific extensions. These extensions make it possible for FT2 to access fonts stored in a font suitcase, something which is unnecessary if you use plain “.ttf” fonts from other sources. Here’s how I did that: Get FT2 and expand the tarball: (in Sources–my version of /SourceCache):

curl -O http://download.savannah.gnu.org/releases/freetype/freetype-2.3.5.tar.gz
cd ..
tar xvfp Sources/freetype-2.3.5.tar.gz

Configure FT2 and make it:

MACOSX_DEPLOYMENT_TARGET=10.5 CFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe -no-cpp-precomp" CCFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe" CXXFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -g -Os -pipe" LDFLAGS="-arch ppc -arch ppc64 -arch i386 -arch x86_64 -bind_at_load" ./configure --with-fsspec=no --with-fsref=no --with-quickdraw-toolbox=no --with-quickdraw-carbon=no
make
sudo make install

Libraries end up in /usr/local/lib.

The funny thing is I have FreeType 2.3.7 courtesy of MacPorts, so I don’t know why the MacPorts installation of PHP is throwing the error unless it is not accessing the MacPorts version of GD. Actually, it makes complete sense. FreeType 2.3.7 in MacPorts is compiled with old font support (which is what triggers the problem). I submitted a trouble ticket requesting a variant of freetype be made available to disable old font support (which won’t work in Leopard anyway).

As I mentioned in my blog post earlier today, I have been having issues using the JpGraph graphing package for PHP with Apple’s built-in PHP under MacOS 10.5. It appears Apple’s security efforts have “secured” PHP to the point where JpGraph (and PDFLib) will not function properly under the built-in PHP. [Note added after initial post: The rest of this post describes installing Apache 2 and PHP under MacPorts and configuring it to be similar to Apple's built-in servers.  This appeared to solve my problems, but then my Mac locked-up [possibly related] and on reboot, the new MacPorts-based PHP installation started throwing the same errors as Apple’s built-in PHP.  More information on this problem is located in my newer post on the issue.]

I had tried to alleviate the solution previously by compiling a version of PHP myself that would be compatible with Apple’s built-in Apache 2 web server. This turned out to be difficult because Apple’s Apache 2 web server is a “universal” binary, meaning it contains four seperate binaries (for 32-bit PowerPC, 64-bit PowerPC, 32-bit Intel, and 64-bit Intel processors). As such I needed to compile a “universal” PHP binary and since I wanted MySQL support, I needed a “universal” MySQL binary. This turned out to be too much for this astronomer, so I gave up on making a new PHP that was compatible with Apple’s built-in web server.

So I bit the bullet and after reviewing the options, I decided to install Apache 2 and PHP under MacPorts. If you have read the other posts on this site, you know I really like MacPorts as the quick and dirty way to get many things running on the Mac. However, despite this, I don’t like installing MacPorts for items Apple provides, instead generally preferring to use Apple’s “pre-installed” versions. Furthermore, there have been a lot of complaints on the MacPorts mailing lists about various issues compiling PHP5. So I wasn’t as quick to jump onto the MacPorts bandwagon for PHP as I am for other problems. However, since I am familiar with MacPorts, I decided this was the best approach for getting my online Finder Charts to work again.

The process proved reasonably painless, especially since I was able to review this blog post, where he lays out pretty much what to do. However, since I wanted to achieve maximum compatibility with Apple’s built-in web server and PHP setup, in case I wanted to switch back, I ended up doing things a little bit differently, so I am outlining my steps here.

1. I started by installing the MacPorts version of Apache 2 using the command sudo port install apache2
2. Next, I had to create the Apache 2 configuration files and edit them. I started by copying the sample configuration file

   sudo cp /opt/local/apache2/conf/httpd.conf.sample /opt/local/apache2/conf/httpd.conf

   and then editing /opt/local/apache2/conf/httpd.conf with my favorite text editor to change the configuration to match the that of the built-in Apache 2 server a closely as possible. My matching the configuration of Apple’s built-in server, I can switch back to it with relative ease if I choose to do so later. So I made the following changes to /opt/local/apache2/conf/httpd.conf:
3. 1. I changed DocumentRoot to "/Library/WebServer/Documents" as is the case with Apple’s built-in server.
   2. To allow index.php files to be used as directory indexes, I changed

      DirectoryIndex index.html

      to

      DirectoryIndex index.html index.php

      (I don’t know why this isn’t necessary with Apple’s built-in server, but it was necessary here).
   3. I changed to and in that directory block of code, I added “MultiViews” to the Options line.
   4. I changed to in order to prevent the listing of .DS_Store files in directory listings by the web server.
   5. Before the ErrorLog block of code in this file, I added the following lines copied from Apple’s default Apache 2 configuration:

      #
      # Apple specific filesystem protection.
      #
      
      Order allow,deny
      Deny from all
      Satisfy All
      
      Order allow,deny
      Deny from all
      Satisfy All

   6. I changed ErrorLog to "/private/var/log/apache2/error_log"
   7. I changed CustomLog to "/private/var/log/apache2/access_log common"
   8. To match Apple’s Apache 2 server configuration, I changed ScriptAliasMatch to

      ^/cgi-bin/((?!(?i:webobjects)).*$) "/Library/WebServer/CGI-Executables/$1"

   9. I changed back to
   10. I added the following Handles to the “To use CGI Scripts” block of code:

       AddHandler imap-file .map
       AddHandler cgi-script .cgi
       AddHandler cgi-script .pl

   11. I uncommended the following lines near the end of the file:

   12. [OPTIONAL] Because I use the WebDAV server on my server, I also uncommented
   13. Finally, I added the following lines to the end of the file in order to allow loading of the PHP5 configuration

4. Next, I editted /opt/local/apache2/conf/extra/httpd-userdir.conf and added the following to the end of the file

   #
   # Users might not be in /Users/*/Sites, so use user-specific config files.
   #
   Include /private/etc/apache2/users/*.conf

5. [OPTIONAL] Since I use the built-in WebDAV server, I made a backup of the WebDAV configuration, then copied the Default MacOS X one, because I have spent a lot of time tweaking it previously and I didn’t want to have to reinvent the wheel. 

   sudo cp /opt/local/apache2/conf/extra/httpd-dav.conf /opt/local/apache2/conf/extra/httpd-dav.conf.orig
   sudo cp /etc/apache2/extra/httpd-dav.conf /opt/local/apache2/conf/extra/httpd-dav.conf

6. I had to install PHP5 with MacPorts. Since I wanted to add support for Apache 2 and MySQL, I entered the command: sudo port install php5 +apache2 +mysql5 +pear which has the side effect of installing MacPorts version of MySQL as well. Assuming everything goes well, after a few minutes (this takes longer than the apache2 install earlier), the installation will end. At this point we can configure the Apache 2 mod_php module by typing: 

   cd /opt/local/apache2/modules/opt/local/apache2/bin/apxs -a -e -n "php5" libphp5.so

7. [OPTIONAL] Next, I had to create the PHP5 configuration file and edit it. I started by copying the sample configuration file sudo cp /opt/local/etc/php.ini-dist /opt/local/etc/php.ini and then editing /opt/local/etc/php.ini to make it match /etc/php.ini (which Apple’s built-in PHP uses). All the changes I made were optional and related to the specifics of my setup. The only interesting one was that I wanted to continue to use the MySQL.com binary distribution of MySQL server, so I set the following variables in /opt/local/etc/php.ini: 
   1. mysql.default_port from “” to 3306
   2. mysql.default_socket from “” to /private/tmp/mysql.sock
   3. mysqli.default_socket from “” to /private/tmp/mysql.sock
8. I deactivated Apple’s built-in web server by turning off Web Sharing in the Sharing.prefPane.
9. Finally, I launched the new webserver (and set it up for launching on boot-up in the future) by typing

   sudo launchctl load -w /Library/LaunchDaemons/org.macports.apache2.plist

   If this breaks anything, I can reverse the process by typing

   sudo launchctl unload -w /Library/LaunchDaemons/org.macports.apache2.plist

10. [OPTIONAL] I like the ability to turn on and off the Apache webserver from the command line using apachectl. I can “emulate” this in /bin/tcsh (my prefered shell) by adding the following command to the ~/.tcshrc file:

    alias apache2ctl 'sudo /opt/local/etc/LaunchDaemons/org.macports.apache2/apache2.wrapper'

    After which I can bring down the server by typing apache2ctl stop and restart it by typing apache2ctl start.

Applying the latest MacOS X Security Update from Apple today reminded me of the problems I have had with PHP on Leopard. The Security Patch brought Leopard up to version 5.2.6. Unfortunately, it doesn’t fix a problem I have been having with PHP5 under Leopard.

While I’m sure Apple did this for security reasons, since upgrading the Leopard, I have had a problem with scripts that use the JpGraph plotting PHP library crashing. After looking in /var/logl/apache2/error_log, I see the following eror message:

The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.

I’ve been trying to figure out why this has been happening for a while. Johan Persson at JpGraph and I exchanged some emails where he outlined for me how this was a problem with Apple’s pre-built version of PHP5 since JpGraph is not fork()-ing any processes. Well, today I decided to look into this again and discovered that the PDFLIb folks have documented this problem a little better, copying from their documentation here:

PHP on Mac OS X. Apple’s PHP version which is bundled with Mac OS X does not work with PDFlib DSOs. To use PHP with PDFlib on Mac OS X you need third-party PHP packages such as MAMP, XAMP[P] for Mac, or Marc Liyanage’s version from www.entropy.ch. Mac OS X 10.5 (Leopard) adds new complications. As described in developer.apple.com/releasenotes/CoreFoundation/CoreFoundation.html it is no longer possible to use CoreFoundation functions after a call to fork( ) without exec( ). However, CoreFoundation functions are required for PDFlib’s host font feature, and the critical sequence above is used in the combination of Apache and PHP. This may trigger the following error message in the Apache log (and can even crash the Apache process):

The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on _THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.

In order to avoid this problem you can run PHP as a CGI on Apache, or disable the host font feature in PDFlib[.]

Sure enough, JpGraph is using host fonts. Damn.

Unfortunately, the Marc Liyanage’s version of PHP5 for Leopard from www.entropy.ch is still in beta and the last update of MAMP was quite a long time ago. And while XAMPP for MacOS X looks promising (at least it has been updated recently), I would really like to keep on using the MySQL server I already have running. So for now, I try installing Apache 2 and PHP using MacPorts.

[See my subsequent posts about getting the MacPorts version of Apache 2 and PHP going and the subsequent continuation of the problem after a reboot.  There is more information about this issue there.]

A few folks have been aware that I am involved in an interesting project to determine the “Shape of our Galaxy”… specifically the nature of the asymmetries observed in the Thick Disk stars in the galaxy. Part of that project involved obtaining multi-fiber spectrograph observations of many stars in selected fields of the sky using the Hectospec multi-fiber spectrometer on the MMT on Mount Hopkins.

Unfortunately, there are very few “external” users of the Hectospec, so the software pipeline available for reducing Hectospec observations, called SPECROAD, was only geared to run on SAO computers. I spent a considerable amount of time last summer and this summer getting that software into a much more portable form, documenting how to use it, and removing bugs from the code.

All this work resulted in the creation of a new version of the Hectospec data pipeline I dubbed “External SPECROAD” or “E-SPECROAD” for short. There were several major problems with the original SPECROAD code that I addressed in E-SPECROAD:

1. Nonportability: The SPECROAD scripts were only designed to run on a Solaris computer at the SAO, with files in specific hard-coded locations using a specific IRAF environment with all the IRAF parameters set before the scripts were run. This has been addressed by trying to assume almost nothing about a user’s IRAF environment from the start except that they have the proper packages installed.
2. Lack of Documentation: The documentation for SPECROAD is frankly inadequate. It was an in-house tool for the SAO, so the publically-available documentation consists mostly of one webpage describing what the scripts do. I have now written up both an installation guide (because there are a lot of pre-requisite pieces of code to install) and a E-SPECROAD user’s guide.
3. No Installation Instructions: There were several IRAF packages to install (available here). The shell scripts required not only the installation of several IRAF packages and the compilation of several C programs, they used korn shell (ksh) which I was unfamiliar with. And, as I would also soon discover, the ksh installed on the Mac had serious bugs in its list handling. There is now a fairly compete installation guide laying out all the pre-requisites and addressing installations on both Macs and Linux boxes.
4. Undocumented Bugs: The available SPECROAD scripts had major bugs whose workarounds were undocumented and only available when I would email someone at the SAO to inquire about them. In one case, you had to know to actually break out of the script with Control-C, then run some of the reduction in IRAF, and then restart the script! E-SPECROAD has been fairly extensively debugged. It’s not bug free, but the exterminator has made some serious passes at the code.
5. Assumed Availability of Spectral Templates: The SPECROAD scripts typically relied on pre-set spectral line databases for different grating/central wavelength configurations instead of having the user exploit the IRAF identify command to build a database. E-SPECROAD is geared toward the user who is going to wavelength calibrate their observations by building the spectral line database for themselves.
6. Assumed Use of Low Resolution Spectra: The SPECROAD scripts were mostly pre-configured for working with 270gpm gratings and required some tweaking for 600gpm. E-SPECROAD can work with either configuration without editing the code.
7. Inconvienent Backups As Data Reduction Progresses: If one of the scripts executed by SPECROAD crashed, you had to dig around subdirectories to “backtrack” the changes before restarting the script. With disk space not an issue, I prefer monolithic backups of the entire data directory at certain key points in the data reduction pipeline. Since you may not like this form of backup (as they consume a lot of disk space), they are optional.

Today, I am posting E-SPECROAD online as a beta release along with installation instructions and a user’s guide. All the other external users of Hectospec I am aware of, all two of you, can enjoy or complain to me about this code. The ones I am unaware of are free to use the code as well.