do i need 32bit libxml2 for python on snow leopard?

Question

i'm having a hell of a time installing scrapy on my sl mbp. it requires libxml2, so i set about installing that. installing it from macports doesn't seem to pull down the python binding.

installing it from source through scrapy's instructions (here) does install the python bindings, but when i run 'python -c "import libxml2"' i get an architecture mismatch:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "libxml2.py", line 1, in <module>
    import libxml2mod
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-    packages/libxml2mod.so, 2): no suitable image found.  Did find:
    /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-    packages/libxml2mod.so: mach-o, but wrong architecture

when i explicitly build the libxml2 dlls to be 32bit, that error goes away, but then libxslt won't build because of some other library not being 32 bit. i'm afraid to keep pulling on that string. so the question is - is python 32bit only? am i doing something stupid here?

edit - this is python 2.6

edit 2 - by popular demand, i'm consolidating @Ned Deily's awesome answer up here. all credit to him, i'm just posting the steps i took based on his response:

if you've been screwing around with mac ports, (and haven't installed anything else through them that you need), nuke them.

$ sudo rm -r /opt/local

add the following to /opt/local/etc/macports/variants.conf to prevent downloading the entire unix library with the next commands

+bash_completion +quartz +ssl +no_x11 +no_neon +no_tkinter +universal +libyaml -scientific

install the macports version of python

$ sudo port install python26

install the pre-reqs for scrapy

sudo port install py26-libxml2 py26-twisted py26-openssl py26-simplejson py26-setuptools python_select
sudo python_select python26

test that the correct python is selected: run $ which python, which should say something along the lines of '/opt/local/bin/python', and that in its own right should be a link to /opt/local/bin/python2.6.

test that the correct architectures are present: run file 'which python' (the single quotes should be backticks, which should spit out (for intel macs running 10.6):

/opt/local/bin/python2.6: Mach-O universal binary with 2 architectures
/opt/local/bin/python2.6 (for architecture x86_64): Mach-O 64-bit executable x86_64
/opt/local/bin/python2.6 (for architecture i386):   Mach-O executable i386

test that libxml installed successfully

$ python -c 'import libxml2'

if all is well, you should get no output.

install scrapy

sudo /opt/local/bin/easy_install-2.6 scrapy

run through the streets (fully clothed) rejoicing that everything is done. again, thanks to ned for the detailed response.

Answer 1

From the paths in your traceback, it appears you have installed and are trying to use the python.org python 2.6.4 (installed to /Library/Frameworks/Python.frameworks ...). That python is 32-bit only. By default on 10.6, MacPorts tries to install 64-bit versions of packages. You can change that for most MacPorts packages by specfying the +universal variant on your MacPorts install commands or by adding it to /opt/local/etc/macports/variants.conf. But, since you need a variety of packages, if you just need this for your own machine you'll likely find it much easier to just install a complete python2.6 64-bit solution using MacPorts. Other than scrapy itself, you should find nearly all of the packages you need already there. You'll need to modify your $PATH to add /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin and /opt/local/bin before any other directories with Python in them. Something like this should work (untested!):

sudo port selfupdate  # ensure you have the latest ports file information
sudo port install py26-libxml2 py26-twisted py26-openssl py26-simplejson py26-setuptools python_select
sudo python_select python26  # optionally make /opt/local/bin/python -> python2.6
sudo /opt/local/bin/easy_install-2.6 scrapy
# or install manually
cd /path/to/scrapy
sudo /opt/local/bin/python2.6 setup.py install

EDIT: For 10.6, due to some issues with lack of Tk support in 64-bit mode, Tk defaults to the X11 version which you probably neither want nor need. I add the following variants to /opt/local/etc/macports/variants.conf for 10.6 (not all of them are applicable to python and friends):

+bash_completion +quartz +ssl +no_x11 +no_neon +no_tkinter +universal +libyaml -scientific

EDIT: If MacPorts is installed properly, you should see something like this on 10.6:

$ ls -l /opt/local/bin/python2.6
lrwxr-xr-x  1 root  wheel  73 Oct 28 20:25 /opt/local/bin/python2.6@ -> /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/python2.6
$ file /opt/local/bin/python2.6
/opt/local/bin/python2.6: Mach-O universal binary with 2 architectures
/opt/local/bin/python2.6 (for architecture x86_64): Mach-O 64-bit executable x86_64
/opt/local/bin/python2.6 (for architecture i386):   Mach-O executable i386

That's if you've added the +universal variant, otherwise you may just see a single x86_64 or i386 architecture.

When using relative paths, make sure your shell $PATH has the two MacPorts directories first:

$ echo $PATH
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin ...

You may need to edit your ~/.bash_profile or other shell file to make that change "permanent". If you want the bare python command to refer to the MacPorts Python, make sure to run the python_select command shown above. If you are still having problems, a nice thing about MacPorts is it is easy to delete and start over again; just:

$ sudo rm -r /opt/local

and download the MacPorts 10.6 installer. But you really shouldn't have to do that. And I think it would really be in your best interests to get this fixed and working as it will likely save you lots of headaches in the future.

Answer 2

Hi

Trying to install but still struggling. When I:

sudo /opt/local/bin/easy_install-2.6 scrapy

I get error message and it stops

AttributeError: 'NoneType' object has no attribute 'get'

Instead I build Scrapy from scratch which works fine.

Then I try what you write

cd /path/to/scrapy
sudo /opt/local/bin/python2.6 setup.py install

Where is /path/to/scrapy? Is that:

/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py

or

/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/Scrapy-0.8-py2.6.egg

or is somewhere else?

Answer 3

Credit to @Ned Deily

These steps seem to work if you want to run Scrapy 0.8 on OS X 10.6. It uses Macports install of Python 2.6 rather than the one bundled with the OS. Steps assume Macports is not installed yet.

Get latest MacPorts installer from here and install:

http://www.macports.org/install.php

sudo port install py26-libxml2 py26-twisted py26-openssl py26-simplejson py26-setuptools python_select

sudo /opt/local/bin/easy_install-2.6 scrapy

Change your ~.profile to:

export     PATH=/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:$PATH

Answer 4

I have a small note to contribute. While attempting to follow-along on this post, I began with three installs of Python:

1. The default Apple Python 2.5.1 located at: /usr/bin/python
2. A version located: /Library/Frameworks/Python.framework/Versions/2.7
3. And a Macport version located: /opt/local/bin/python2.6

My trouble was that:

$ python

would always default to the 2.7 when I needed it to use the Macports version. The following did not help:

$ sudo python_select python26

I even removed the 2.7 version which caused only an error.

I figured out I needed to change the default path to the Macports version using the following:

$ PATH=$PATH\:/opt/local/bin ; export PATH

And then reinitiate the ports, etc.

Finally, I was not able to reference the scrapy-ctl.py file by default through these instructions so I had to reference the scrapy-ctl.py file directly

/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py

UPDATE

A quick addendum to this post with instructions to create the link, found on the Scrapy site (#2 and #3).

Starting with #2, "Add Scrapy to your Python Path"

sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scrapy

And #3, "Make the scrapy command available"

sudo ln -s /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/scrapy-ctl.py /usr/local/bin/scrapy