Compare many text files that contain duplicate "stubs" from the previous and next file and remove duplicate text automatically

Answer 1

Are the stubs identical to the end of the previous file? Or different line endings/OCR mistakes?

Is there a way to discern an article's beginning? Maybe an indented abstract? Then you could go through each file and discard everything before the first and after (including) the second title.

Answer 2

Post a sample file, or at least the top and bottom of a file so we can see what we're dealing with. An example is worth 1000^2 words.

Answer 3

Are the titles & author always on a single line? And does that line always contain the word "BY" in uppercase? If so, you can probably do a fair job withn awk, using those criteria as the begin/end marker.

Edit: I really don't think that using diff is going to work as it is a tool for comparing broadly similar files. Your files are (from diff's point of view) actually completely different - I think it will get out of sync immediately. But then, I'm not a diff guru :-)

Answer 4

You should probably try something like this (I've now tested it on the sample data you provided):

#!/usr/bin/ruby

class A_splitter
    Title   = /^[A-Z]+[^a-z]*$/
    Byline  = /^BY /
    Number = /^\d*$/
    Blank_line = /^ *$/
    attr_accessor :recent_lines,:in_references,:source_glob,:destination_path,:seen_in_last_file
    def initialize(src_glob,dst_path=nil)
        @recent_lines = []
        @seen_in_last_file = {}
        @in_references = false
        @source_glob = src_glob
        @destination_path = dst_path
        @destination = STDOUT
        @buffer = []
        split_em
        end
    def split_here
        if destination_path
            @destination.close if @destination
            @destination = nil
          else
            print "------------SPLIT HERE------------\n" 
          end
        print recent_lines.shift
        @in_references = false
        end
    def at_page_break
        ((recent_lines[0] =~ Title  and recent_lines[1] =~ Blank_line and recent_lines[2] =~ Number) or
         (recent_lines[0] =~ Number and recent_lines[1] =~ Blank_line and recent_lines[2] =~ Title))
        end
    def print(*args)
        (@destination || @buffer) << args
        end
    def split_em
        Dir.glob(source_glob).sort.each { |filename|
            if destination_path
                @destination.close if @destination
                @destination = File.open(File.join(@destination_path,filename),'w')
                print @buffer
                @buffer.clear
              end
            in_header = true
            File.foreach(filename) { |line|
                line.gsub!(/\f/,'')
                if in_header and seen_in_last_file[line]
                    #skip it
                  else 
                    seen_in_last_file.clear if in_header
                    in_header = false
                    recent_lines << line
                    seen_in_last_file[line] = true
                  end
                3.times {recent_lines.shift} if at_page_break
                if recent_lines[0] =~ Title and recent_lines[1] =~ Byline
                    split_here
                  elsif in_references and recent_lines[0] =~ Title and recent_lines[0] !~ /\d/
                    split_here
                  elsif recent_lines.length > 4
                    @in_references ||= recent_lines[0] =~ /^REFERENCES *$/
                    print recent_lines.shift
                  end
                }
            } 
        print recent_lines
        @destination.close if @destination
        end
    end

A_splitter.new('bul_*_*_*.txt','test_dir')

Basically, run through the files in order, and within each file run through the lines in order, omitting from each file the lines that were present in the preceding file and printing the rest to STDOUT (from which it can be piped) unless a destination director is specified (called 'test_dir' in the example see the last line) in which case files are created in the specified directory with the same name as the file which contained the bulk of their contents.

It also removes the page-break sections (journal title, author, and page number).

It does two split tests:

• a test on the title/byline pair
• a test on the first title-line after a reference section

(it should be obvious how to add tests for additional split-points).

Retained for posterity:

If you don't specify a destination directory it simply puts a split-here line in the output stream at the split point. This should make it easier for testing (you can just less the output) and when you want them in individual files just pipe it to csplit (e.g. with

csplit -f abstracts - '---SPLIT HERE---' '{*}'

or something) to cut it up.

Answer 5

Here's is the beginning of another possible solution in Perl (It works as is but could probably be made more sophisticated if needed). It sounds as if all you are concerned about is removing duplicates across the corpus and don't really care if the last part of one article is in the file for the next one as long as it isn't duplicated anywhere. If so, this solution will strip out the duplicate lines leaving only one copy of any given line in the set of files as a whole.

You can either just run the file in the directory containing the text files with no argument or alternately specify a file name containing the list of files you want to process in the order you want them processed. I recommend the latter as your file names (at least in the sample files you provided) do not naturally list out in order when using simple commands like ls on the command line or glob in the Perl script. Thus it won't necessarily compare the correct files to one another as it just runs down the list (entered or generated by the glob command). If you specify the list, you can guarantee that they will be processed in the correct order and it doesn't take that long to set it up properly.

The script simply opens two files and makes note of the first three lines of the second file. It then opens a new output file (original file name + '.new') for the first file and writes out all the lines from the first file into the new output file until it finds the first three lines of the second file. There is an off chance that there are not three lines from the second file in the last one but in all the files I spot checked that seemed to be the case because of the journal name header and page numbers. One line definitely wasn't enough as the journal title was often the first line and that would cut things off early.

I should also note that the last file in your list of files entered will not be processed (i.e. have a new file created based off of it) as it will not be changed by this process.

Here's the script:

#!/usr/bin/perl
use strict;

my @files;
my $count = @ARGV;
if ($count>0){
    open (IN, "$ARGV[0]");
    @files = <IN>;
    close (IN);
} else {
    @files = glob "bul_*.txt";
}
$count = @files;
print "Processing $count files.\n";

my $lastFile="";
foreach(@files){
    if ($lastFile ne ""){
     print "Processing $_\n";
     open (FILEB,"$_");
     my @fileBLines = <FILEB>;
     close (FILEB);
     my $line0 = $fileBLines[0];
            if ($line0 =~ /\(/ || $line0 =~ /\)/){
                    $line0 =~ s/\(/\\\(/;
                    $line0 =~ s/\)/\\\)/;
            }
     my $line1 = $fileBLines[1];
     my $line2 = $fileBLines[2];
     open (FILEA,"$lastFile");
     my @fileALines = <FILEA>;
     close (FILEA);
     my $newName = "$lastFile.new";
     open (OUT, ">$newName");
     my $i=0;
     my $done = 0;
     while ($done != 1 and $i < @fileALines){
      if ($fileALines[$i] =~ /$line0/ 
       && $fileALines[$i+1] == $line1
       && $fileALines[$i+2] == $line2) {
       $done=1;
      } else {
       print OUT $fileALines[$i];
       $i++;
      }
     }
     close (OUT);
    }
    $lastFile = $_;
}

EDIT: Added a check for parenthesis in the first line that goes into the regex check for duplicity later on and if found escapes them so that they don't mess up the duplicity check.

Answer 6

A quick stab at it, assuming that the stub is strictly identical in both files:

#!/usr/bin/perl

use strict;

use List::MoreUtils qw/ indexes all pairwise /;

my @files = @ARGV;

my @previous_text;

for my $filename ( @files ) {
    open my $in_fh,  '<', $filename          or die;
    open my $out_fh, '>', $filename.'.clean' or die;

    my @lines = <$in_fh>;
    print $out_fh destub( \@previous_text, @lines );
    @previous_text = @lines;
}


sub destub {
    my @previous = @{ shift() };
    my @lines = @_;

    my @potential_stubs = indexes { $_ eq $lines[0] } @previous;

    for my $i ( @potential_stubs ) {
        # check if the two documents overlap for that index
        my @p = @previous[ $i.. $#previous ];
        my @l = @lines[ 0..$#previous-$i ];

        return @lines[ $#previous-$i + 1 .. $#lines ]
                if all { $_ } pairwise { $a eq $b } @p, @l;

    }

    # no stub detected
    return @lines;
}

Answer 7

It looks like a much simpler solution would actually work.

No one seems to be using the information provided by the filenames. If you do make use of this information, you may not have to do any comparisons between files to identify the area of overlap. Whoever wrote the OCR probably put some thought into this problem.

The last number in the file name tells you what the starting page number for that file is. This page number appears on a line by itself in the file as well. It also looks like this line is preceded and followed by blank lines. Therefore for a given file you should be able to look at the name of the next file in the sequence and determine the page number at which you should start removing text. Since this page number appears in your file just look for a line that contains only this number (preceded and followed by blank lines) and delete that line and everything after. The last file in the sequence can be left alone.

Here's an outline for an algorithm

1. choose a file; call it: file1
2. look at the filename of the next file; call it: file2
3. extract the page number from the filename of file2; call it: pageNumber
4. scan the contents of file1 until you find a line that contains only pageNumber
5. make sure this line is preceded and followed by a blank line.
6. remove this line and everything after
7. move on to the next file in the sequence