How can I find the difference in 2 data sets?

Question

If I have 2 pipe delimited files containing bookmark data, for example. How can I read in the data then determine the difference in the two sets of data?

Input Set #1: bookmarks.csv

2|www.cnn.com|News|This is CNN
3|www.msnbc.com|Search|
4|news.ycombinator.com|News|Tech News
5|bing.com|Search|The contender

Input Set #2: bookmarks2.csv

1|www.google.com|Search|The King of Search
2|www.cnn.com|News|This is CNN
3|www.msnbc.com|Search|New Comment
4|news.ycombinator.com|News|Tech News

Output

Id #1 is missing in set #1
Id #5 is missing in set #2
Id #3 is different:
 ->www.msnbc.com|Search|
 ->www.msnbc.com|Search|New Comment

Answer 1

Run it through a diff?

Answer 2

put the first data in a dictionary (hashtable) with the id as key

the read the next data line by line, retrieve the id from the hash.

• if the id is not in the hash, output: id missing in set 1
• if the value in the has differs, output: id is different
• store the id's in a second hashtable

then run through the keys of the first hashtable

• check if they are also in the second hashtable. if not output: id is missing in set2

Answer 3

split them with re regexp and make a set out of them with (apply set (re-seq ... ) then call (difference set1 set2) to find the things that are in set 1 and not set 2. reverse it to find this items in set 2 that are not in set one.

look at http://clojure.org/data%5Fstructures for more info on clojure sets.

Answer 4

Here is my stab at a functional-ish approach to the problem:

• Create 2 maps, one for each file
• Find the missing items between the two maps, using dissoc
• Find the different, but shared items between the two maps, using intersection and filter

Code

(ns diffset
  (:use [clojure.contrib.duck-streams]
        [clojure.set]))

(def file1 "bookmarks.csv")
(def file2 "bookmarks2.csv")

(defn split-record [line]
  "split line into (id, bookmark)"
  (map #(apply str %)
       (split-with #(not (= % \|)) line)))

(defn map-from-file [f]
  "create initial map from file f"
  (with-open [r (reader f)]
    (doall (apply hash-map (apply concat (map split-record
                                              (line-seq r)))))))

(defn missing [x y]
  "return seq of all ids in x that are not in y"
  (keys (apply dissoc x (keys y))))

(defn different [x y]
  "return seq of all ids that match but have different bookmark string"
  (let [match-keys (intersection (set (keys x)) (set (keys y)))]
    (filter #(not (= (get x %)
                     (get y %)))
            match-keys)))

(defn diff [file1 file2]
  "print out differences between two bookmark files"
  (let [[s1 s2] (map map-from-file [file1 file2])]
    (dorun (map #(println (format "Id #%s is missing in set #1" %))
                (missing s2 s1)))
    (dorun (map #(println (format "Id #%s is missing in set #2" %))
                (missing s1 s2)))
    (dorun (map #(println (format "Id #%s is different:" %) "\n"
                          " ->" (get s1 %) "\n"
                          " ->" (get s2 %)) (different s1 s2)))))

Result

user> (use 'diffset)
nil
user> (diff file1 file2)
Id #1 is missing in set #1
Id #5 is missing in set #2
Id #3 is different: 
  -> |www.msnbc.com|Search| 
  -> |www.msnbc.com|Search|New Comment
nil

Answer 5

(use '[clojure.contrib str-utils duck-streams pprint]
     '[clojure set])

(defn read-bookmarks [filename]
  (apply hash-map
         (mapcat #(re-split #"\|" % 2)
                 (read-lines filename))))

(defn diff-bookmarks [filename1 filename2]
  (let [f1 (read-bookmarks filename1)
        f2 (read-bookmarks filename2)
        k1 (set (keys f1))
        k2 (set (keys f2))
        missing-in-1 (difference k2 k1)
        missing-in-2 (difference k1 k2)
        present-but-different (filter #(not= (f1 %) (f2 %))
                                      (intersection k1 k2))]
    (cl-format nil "~{Id #~a is missing in set #1~%~}~{Id #~a is missing in set #2~%~}~{~{Id #~a is different~%  -> ~a~%  -> ~a~%~}~}"
               missing-in-1
               missing-in-2
               (map #(list % (f1 %) (f2 %))
                    present-but-different))))

(print (diff-bookmarks "bookmarks.csv" "bookmarks2.csv"))