Streaming and Linq Blobs

views:

1112

answers:

+4 Q:

Streaming and Linq Blobs

I have an object I am using to store document meta data into a table. The body text of the document can be very large, sometimes > 2GB so I will be storing it into a nvarchar(max) field in SQL 2008. I'll use SQL 2008 later to index that field. I won't be using filestreams because they are very restrictive to the database and prevents certain types of concurrency locking schemes.

This object is exposed to the developer via LinqToSQL. My concern is that the field will be to large and I have seen .Net bomb out with an OutOfMemory exception if the text is > 1.5 GB.

So I am wondering, can I treat this blob as a stream with Linq? Or do I have to bypass Linq altogether if I want to use a blob?

+2 A:

Given the answer to "Can a LINQ query retrieve BLOBs [...]" I suspect you're out of luck. The System.Data.Linq.Binary type doesn't have any mechanism for streaming - basically it's just an immutable byte array representation.

There may be some deep LINQ mojo you could invoke, but I suspect it would really have to be pretty deep.

It's possible that the Entity Framework would handle it - I haven't investigated that.

Jon Skeet 2008-11-14 08:43:35

Thanks for the feedback. I was thinking maybe I could add a new property to deal with the blob field and extend the partial class to stream the binary data to the database separate from Linq. Then the developer using the class won't be able to tell the difference.

Jiyosub 2008-11-14 08:48:49

I suspect it won't be quite as seamless as that (I suspect that projections using the property won't work), but that sounds mostly feasible. Do you need to write to the database from this app, or just read? Reading is likely to be a *lot* easier :)

Jon Skeet 2008-11-14 08:50:44

Alternatively, writing "outside LINQ" should be fine - it's just the developer won't be able to set the property to a stream and call SubmitChanges to get it all there. At least, not without a lot of work from you, I suspect.

Jon Skeet 2008-11-14 08:51:39

I actually have covered the entire entity layer with my own classes, so I can use Linq, Entity Framework, Hibernate, etc without changing any code calling my entities. So maybe I'll have to create some interface IStream or something which allows those attributes to be streamed to the DB.

Jiyosub 2008-11-14 08:55:56

+1 A:

I ended up writing my own method around linqtoSql utlising the write method avaiable to varchar(max) objects in SQL. This allows developers to chunk inserts into the DB for large data types.

Jiyosub 2008-12-08 03:03:36

Can you give more details about your solution, I am stuck with the same problem?

Mart 2009-12-23 12:38:52

ansaurus

tags:

views:

answers:

Streaming and Linq Blobs

related questions