Saving nonprintable characters to the database in Rails

Question

I am unable to save nonprintable characters (e.g. "\x83") to the database using Rails (v2.2).

Simplified example (from the console):

>> f = FileSpace.create( { :name => "/tmp/\x83" } )
=> #<FileSpace id: 7, name: "/tmp/\203">
>> f.name
=> "/tmp/\203"
>> FileSpace.last
=> #<FileSpace id: 7, name: "/tmp/">

So you can see, Rails is silently discarding the "\x83" (or "\203") character from the string.

The "\83" character is not stored in the database:

mysql> SELECT hex(name) FROM file_spaces WHERE id=7;
+------------------------------------+
| hex(name)                          |
+------------------------------------+
| 2F746D702F                         | 
+------------------------------------+
1 rows in set (0.03 sec)

mysql> select x'2F746D702F';
+---------------+
| x'2F746D702F' |
+---------------+
| /tmp/         | 
+---------------+
1 row in set (0.00 sec)

So, how can I get Rails to save the nonprintable character?

Answer 1

I can't tell you where exactly \x83 is lost, but my guess would be the database (at least PostgreSQL rejects that string because of invalid byte encoding).

To work arround you could base64 encode your string and store that one:

require 'base64'
str = "/tmp/\x83"
encoded = Base64.encode64(str) => "L3RtcC+D\n"
Base64.decode64(encoded) => "/tmp/\x83"

Answer 2

The solution I ended up with was inspired by both @dsander's answer and @Glex's comment, and uses callbacks. I had to first create a name_encoded field (default: false) for the file_spaces table in the database, because already-saved file spaces are not encoded.

Then, I created a model to use for callbacks (unfortunately, not the cleanest code):

class EncodingWrapper
  require 'base64'

  def initialize(attribute)
    @attribute = attribute.to_s
    @get_method = @attribute
    @set_method = @attribute + "="
    @get_encoded_method = @attribute + "_encoded"
    @set_encoded_method = @attribute + "_encoded="
  end

  def before_save(record)
    set_encoded(record)
    encode(record)
  end

  def after_save(record)
    decode(record)
  end

  # Rails dislikes the after_find callback because it has a potentially
  # severe performance penalty.  So it ignores it unless it is created a
  # particular way in the model.  So I have to change the way this method
  # works.  So long, DRY code. :-/
  def self.after_find(record, attribute)
    # Ugly...but hopefully relatively fast.
    a = attribute.to_s
    if record.send(a + '_encoded')
      record.send(a + '=', Base64.decode64(record.send(a)))
    end
  end

  private

  def is_encoded?(record)
    record.send(@get_encoded_method)
  end

  def set_encoded(record)
    record.send(@set_encoded_method, true)
  end

  def encode(record)
    record.send(@set_method, Base64.encode64(record.send(@get_method)))
  end

  def decode(record)
    record.send(@set_method, Base64.decode64(record.send(@get_method)))
  end

end

Last, hook the callbacks into the FileSpace model:

class FileSpace < ActiveRecord::Base
  ...
  before_save EncodingWrapper.new(:name)
  after_save  EncodingWrapper.new(:name)
  # Have to do the after_find callback special, to tell Rails I'm willing to
  # pay the performance penalty for this feature.
  def after_find
    EncodingWrapper.after_find(self, :name)
  end
  ...
end