Postgresql causing "no space on device" error but disks/partitions are not full

Question

I'm getting the following error while running my Rails website and I believe it is causing it to go down. I'm using Passenger and my site is heavily trafficked. When I try to bring it back online, Apache spawns a large number of children and the load on the machine spikes to about 50, and the site becomes extremely slow. I have no shortage of disk.

PGError (FATAL:  could not access status of transaction 0
DETAIL:  could not write to file "pg_subtrans/009B" at offset 8192: No space left on device):
    /vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:876:in `initialize'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:876:in `connect'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:876:in `connect'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:276:in `initialize'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `new'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb:292:in `send'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb:292:in `connection='
    /vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb:260:in `retrieve_connection'
    /vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/connection_specification.rb:78:in `connection'
    /vendor/rails/activerecord/lib/active_record/query_ca

Answer 1

The "No space left on device" error isn't necessarily caused by running out of space. It can also be caused by your filesystem's inode table filling up - in other words, your filesystem can only hold so many files, and you've reached that limit.

This can happen if you have a very large number of tiny files. In these situations it's common to build a new filesystem with a large inode table, to allow for a larger number of small files.

Answer 2

Another possibility would be per-user disk quotas - see http://www.postgresql.org/docs/8.4/static/disk-full.html