I recently read a nice post on using StringIO in Ruby. What the author doesn't mention, though, is that StringIO is just an "I." There's no "O." You can't do this, for example: s = StringIO.new s << 'foo' s << 'bar' s.to_s # => should be "foo\nbar" # => really is ''` Ruby really needs a StringBuffer just like the one Java has. Stri...
In Python you can use StringIO for a file-like buffer for character data. Memory-mapped file basically does similar thing for binary data, but it requires a file that is used as the basis. Does Python have a file object that is intended for binary data and is memory only, equivalent to Java's ByteArrayOutputStream? The use-case I have i...
I currently use the following function to do a simple HTTP GET. public static String download(String url) throws java.io.IOException { java.io.InputStream s = null; java.io.InputStreamReader r = null; //java.io.BufferedReader b = null; StringBuilder content = new StringBuilder(); try { s = (java.io.InputStrea...
I am writing a program in dot net that will execute scripts and command line programs using the framework 2.0's Process object. I want to be able to access the screen buffers of the process in my program. I've investigated this and it appears that I need to access console stdout and stderr buffers. Anyone know how this is accomplished us...
I am sequentially processing a large file and I'd like to keep a large chunk of it in memory, 16gb ram available on a 64 bit system. A quick and dirty way is to do this, is simply wrap the input stream into a buffered input stream, unfortunately, this only gives me a 2gb buffer. I'd like to have more of it in memory, what alternatives d...
I am using VB6 and the Win32 API to write data to a file, this functionality is for the export of data, therefore write performance to the disk is the key factor in my considerations. As such I am using the FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH options when opening the file with a call to CreateFile. FILE_FLAG_NO_BUFFERING ...
Hello, I use this code to create a .zip with a list of files: ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(zipFile)); for (int i=0;i<srcFiles.length;i++){ String fileName=srcFiles[i].getName(); ZipEntry zipEntry = new ZipEntry(fileName); zos.putNextEntry(zipEntry); InputStream fis = new FileInputStream...
I have a method that creates a MessageDigest (a hash) from a file, and I need to do this to a lot of files (>= 100,000). How big should I make the buffer used to read from the files to maximize performance? Most everyone is familiar with the basic code (which I'll repeat here just in case): MessageDigest md = MessageDigest.getInstance(...
Isn't the point of caching things in main memory to avoid the expensive disk i/o? If you're caching things in the swap space of a hard drive, how does that avoid disk i/o? ...
I need a java api, that cannot use JMF, to play video interpreted by the SO codecs but i want to retrieve the each frames in java code. Somebody know some? ...
I have to deal with very large text files (2 GBs), it is mandatory to read/write them line by line. To write 23 millions of lines using ofstream is really slow so, at the beginning, I tried to speed up the process writing large chunks of lines in a memory buffer (for example 256 MB or 512 MB) and then write the buffer into the file. This...
We need to develop some kind of buffer management for an application we are developing using C#. Essentially, the application receives messages from devices as and when they come in (there could be many in a short space of time). We need to queue them up in some kind of buffer pool so that we can process them in a managed fashion. We w...
Hi There is a question I have been wondering about for ages and I was hoping someone could give me an answer to rest my mind. Let's assume that I have an input stream (like a file/socket/pipe) and want to parse the incoming data. Let's assume that each block of incoming data is split by a newline, like most common internet protocols. T...
Hi There, I am reading log files but not all lines want to be processed straight away. I am using a queue / buffer to store the lines while they wait to be processed. This queue is regularly scanned for particular lines - when they are found, they are removed from the queue (they can be anywhere in it). When there isn't a particular li...
Is there any reason to prefer a CharBuffer to a char[] in the following: CharBuffer buf = CharBuffer.allocate(DEFAULT_BUFFER_SIZE); while( in.read(buf) >= 0 ) { out.append( buf.flip() ); buf.clear(); } vs. char[] buf = new char[DEFAULT_BUFFER_SIZE]; int n; while( (n = in.read(buf)) >= 0 ) { out.write( buf, 0, n ); } (where in...
I need a byte buffer class in Java for single-threaded use. I should be able to insert data at the back of the buffer and read data at the front, with an amortized cost of O(1). The buffer should resize when it's full, rather than throw an exception or something. I could write one myself, but I'd be very surprised if this didn't exist y...
Building on http://stackoverflow.com/questions/318553/getting-emacs-to-untabify-when-saving-files , I'd like to run a hook to untabify my C++ files when I start modifying the buffer. I tried adding hooks to untabify the buffer on load, but then it untabifies all my writable files that are autoloaded when emacs starts. (For those that wo...
I don't really get it and it's driving me nuts. i've these 4 lines: Image img = Image.FromFile("F:\\Pulpit\\soa.bmp"); MemoryStream imageStream = new MemoryStream(); img.Save(imageStream, ImageFormat.Bmp); byte[] contentBuffer = new byte[imageStream.Length]; imageStream.Read(contentBuffer, 0, contentBuffer.Length); when debugging i ca...
I keep a cache of transactions to flush (to persistent storage) on the event of a watermark or object finalization. Since __del__ is no longer guaranteed to be called on every object, is the appropriate approach to hook a similar function (or __del__ itself) into atexit.register (during initialization)? If I'm not mistaken, this will ca...
this seems so silly - i must be missing something obvious. I have the following code (just as a test): <%@ Page Language="C#" %> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <script runat="server"> void page_load(object o, EventArgs e) { Re...