Remove Duplicate Line in Vim?

Question

I'm trying to use VIM to remove a duplicate line in an XML file I created. (I can't recreate the file because the ID numbers will change.)

The file looks something like this:

    <tag k="natural" v="water"/>
    <tag k="nhd:fcode" v="39004"/>
    <tag k="natural" v="water"/>

I'm trying to remove one of the duplicate k="natural" v="water" lines. When I try to use the \_ modifier to include newlines in my regex replaces, VIM doesn't seem to find anything.

Any tips on what regex or tool to use?

Answer 1

Are you trying to search and replace the line with nothing? You could try the g command instead:

:%g/search_expression_here/d

The d at the end tells it to delete the lines that match.

You may find more tips here.

Answer 2

instead of using vim you do something like

sort filename | uniq -c | grep -v "^[ \t]*1[ \t]"

to figure out what is the duplicate line and then just use normal search to visit it and delete it

Answer 3

This link addresses the problem

Answer 4

Answers using 'uniq' suffer from the problem that 'uniq' only finds adjacent duplicated lines, or the data file is sorted losing positional information.

If no line may ever be repeated, then it is relatively simple to do in Perl (or other scripting language with regex and associative array support), assuming that the data source is not incredibly humungous:

#!/bin/perl -w
# BEWARE: untested code!
use strict;
my(%lines);
while (<>)
{
    print if !defined $lines{$_};
    $lines{$_} = 1;
}

However, if it is used indiscriminately, this is likely to break the XML since end tags are legitimately repeated. How to avoid this? Maybe by a whitelist of 'OK to repeat' lines? Or maybe only lines with open tags with values are subject to duplicate elimination:

#!/bin/perl -w
# BEWARE: untested code!
use strict;
my(%lines);
while (<>)
{
    if (m%^\s*<[^\s>]+\s[^\s>]+%)
    {
         print if !defined $lines{$_};
         $lines{$_} = 1;
    }
    else
    {
         print;
    }
}

Of course, there is also the (largely valid) argument that processing XML with regular expressions is misguided. This coding assumes the XML comes with lots of convenient line breaks; real XML may not contain any, or only a very few.

Answer 5

You could select the lines then do a :'<,'>sort u if you don't care about the ordering. It will sort and remove duplicates.

Answer 6

to the OP, if you have bash 4.0

#!/bin/bash
# use associative array
declare -A DUP
file="myfile.txt"
while read -r line
do
    if [ -z ${DUP[$line]} ];then
        DUP[$line]=1
        echo $line >temp
    fi
done < "$file"
mv temp "$file"

Answer 7

with python to remove all repeated lines:

#!/usr/bin/env python

import sys
def remove_identical(filein, fileout) : 
  lines = list()
  for line in open(filein, 'r').readlines() : 
    if line not in lines : lines.append(line)
  fout = open(fileout, 'w')
  fout.write(''.join(lines))
  fout.close()

remove_identical(sys.argv[1], sys.argv[2])