I'm working on a solution where I need to associate metadata with files. In order to be able to associate the right file with the right metadata if the file is moved for instance I need to be able create a "fingerprint" of sorts to identify the file. The obvious solution would be simply to calculate a hash from the file contents, howev...
Hello, I am writing a utility which accepts either a filename, or reads from stdin I would like to know the most robust / fastest way of checking to see if stdin exists (data is being piped to the program) and if so reading that data in. If it doesn't exist, the processing will take place on the filename given. I have tried using the...
I am trying to read binary data from sys.stdin using Python 2.7 on Windows XP. The binary data is a WAV file decoded by foobar2000. Normally, this data is sent to a command-line encoder such as lame.exe on stdin, where it is processed and written to an output file whose name is provided in the command-line arguments. I am trying to inter...
Hey all, Here is my situation. I would like to make writing to the file system as efficient as possible in my application. The app is multi-threaded and each thread can possibly write to the same file. Is there a way that I can write to the file asynchronously from each thread without having the writes in the different threads bang h...
Hi! I have a few syncing routines that I'd like to use for FTP uploads. However they all use the functions in the IO namespace, and I was wondering whether I could use it to access distant files stored on an FTP server. Maybe should I mount the distant server as a network drive? Is there a way to do this programatically? Thanks, CFP. ...
Is there something like the sendfile-syscall that works with multiple target file descriptors (i.e. instead of copying from one FD to another FD, it could should copy to, say, 4 FDs)? I know that when talking about asynchronous IO, this is known as gather/scatter, but I could not find anything in the Linux AIO documentation. ...
I'm playing with this code on Linux 2.6.16.46: io.aio_fildes = open(name, O_CREAT | O_TRUNC | O_WRONLY | O_SYNC, 00300); io.aio_buf = buffer; io.aio_nbytes = size; io.aio_sigevent = sigev; io.aio_lio_opcode = LIO_WRITE; aio_write( &io ); This should use the memory pointed by buffer for the IO operation. Still, I see the number of di...
Hello everyone! I would like to use the same code for copying files to an FTP server, and through USB. I've already written some code, which as of now uses functions from the System.IO namespace. To minimize code duplication, I've listed the functions which I need from the system.IO namespace, and I was thinking of designing an interfa...
This works just fine in my dev environment (I am rewriting a css file): File.open(RAILS_ROOT + '\public\stylesheets\colors.css', 'w') do |w| w.puts 'some_text' end But when I run it in my prod environment (on Dreamhost) nothing happens - the file is not modified - nothing. What I need to be able to do is to overwrite an exist...
I am trying to read a csv file and I am getting the error above but the file is there. The line giving the error is infilequery = file('D:\x88_2.csv','rb') and I get the error below. Traceback (most recent call last): File "C:\Python26\usrapply_onemol2.py", line 14, in infilequery = file('D:\x88_2.csv','rb') IOError: [Errn...
The JDK docs say, that if a thread is interrupted that currently blocks in an io operation of an InterruptibleChannel, the channel is closed and a ClosedByInterruptException is thrown. However, i get a different behaviour when using a FileChannel: public class Main implements Runnable { public static void main(String[] args) throws Exc...
I allocated some memory with anonymous mmap: buff->addr = mmap(NULL, length, PROT_NONE, MAP_PRIVATE | MAP_ANONYMOUS -1, 0); fprintf(stderr, "allocated buffer: %p, %lu\n", buff->addr, (unsigned long)length); then I'm writing to it using fd opened with O_DIRECT: int fd = open(name, O_CREAT | O_TRUNC | O_WRONLY | O_DIRECT, 00300); if(fd...
I am running Django under Apache+mod_wsgi in daemon mode with the following config: WSGIDaemonProcess myserver processes=2 threads=15 My application does some IO on the backend, which could take several seconds. def my_django_view: content=... # Do some processing on backend file return HttpResponse(content) It appears tha...
In our project sometimes when we use InputStream.read(byte[] b) method, there is some error. When we use byte[] b = new byte[1024]; int len = -1; while ((len = io.read(b, 0, 1024)) != -1) { response.getOutputStream().write(b, 0, len); } then it goes well. I found the source code, which is amazing public int read(byte b[]) thro...
I would like to gather amount of time my application is waiting for I/O. I am running this java application on ubuntu/linux. I am using yourkit profiler. Suggest if there any other profiling tool for measuring I/O time. ...
I have tried to do it myself--asking for help on specific functions, but the more I think about the possibilities, the more lost I get. I have some software (quantum chemistry packages). That reads input files and generates output files that are basically clumps of data of the form: Energy:[many spaces]6.3432 H 5 O 33 OHO 32 And weird st...
Just to be clear, I'm not looking for the MIME type. Let's say i have the input: "/path/to/file/foo.txt" I'd like a way to break this input up, specifically into ".txt" for the extension. Is there any built in ways to do this in Java? I'm still new and would like to avoid writing my own parser... :) Thanks! ...
Currently my application takes in a text file/files, parses them into another file type and puts them on disk. I then call a secondary program (not mine) to process THAT text file into a third. The foreign program basically does one thing: program.exe --input:foo.txt --output:bar.txt What I would like to know... could I replace th...
Hi all I want to read a very big text file(a log file of a web app)and do some processing. Is there any Framework to help doing such work ? The file is 100M+,shall I use mutil-thread ? best regards ...
I've been looking through a lot of code made by others lately and happened to notice everyone uses "printf" style C functions a lot, but the C++ functions learned in school (cout, specifically) don't seem so popular. Is this a valid observation, and is there a reason for this? Convention? Thanks, R ...