The code in the question is trying to define class methods based on instance variables. There's no way this will work. Also :price is just a symbol. It can never have any value beyond it's internal value. The following results in syntax errors:
:price = nil
:price = anything
Symbols are always defined and have an internal value, which is set to the text the string after the ':'. Meaning :price.nil?
and :price == nil
will never be true. You're getting confused because you see :price passed to other class methods. What's really happening is that those methods take the symbol and dynamically turn into an instance method name to be called, a column name when executed, or just uses it as a key in a hash.
Enough about why what you were doing doesn't work. Let's talk about how to fix it.
You want to use the if and unless clauses for validations. This code does exactly what you want.
# app/models/item.rb
validates_numericality_of :cost, :greater_than => 0
validates_numericality_of :quantity, :greater_than => 0,
:if => Proc.new {|item| item.price.nil?}
validates_numericality_of :price, :greater_than => 0,
:unless => Proc.new {|item| item.price.nil?}
end