duplicates

Java Scanner class reading strings

I got the following code: int nnames; String names[]; System.out.print("How many names are you going to save: "); Scanner in = new Scanner(System.in); nnames = in.nextInt(); names = new String[nnames]; for (int i = 0; i < names.length; i++){ System.out.print("Type a name: "); names[i] = ...

Make MySQL table unique

Hay, I created a spider to crawl through a PDF document and log every word in the document into a table in a MySQL database. Obviously words like 'the', 'and', 'or' etc appear in a book many, many times. I'm just wondering what's the quickest method to remove dupe values from a table? ...

sql query distinct on multiple columns

i have this data and i am trying to find cases where there are different ids but duplicate data in Field 1,2,3,4 id field1 field2 field3 field4 ==== ====== ====== ===== ======= 1 A B C D 2 A B C D 3 A A C B 4 A A C B so, in whatever ...

Duplicate Insertions in Database using sqlite, sqlalchemy, python

I am learning Python and, through the help of online resources and people on this site, am getting the hang of it. In this first script of mine, in which I'm parsing Twitter RSS feed entries and inserting the results into a database, there is one remaining problem that I cannot fix. Namely, duplicate entries are being inserted into one o...

Help with ON DUPLICATE KEY UPDATE implementation/design

I have a table, sessionBasket, that holds a list of the items in the shopping baskets of visitors to my site. It looks like: id INT NOT NULL AUTO_INCREMENT PRIMARY KEY usersessid VARCHAR date_added DATETIME product_id INT qty INT My add to basket script first checks for the presence of an item with the current product_id in this table ...

Algorithm: efficient way to remove duplicate integers from an array

I got this problem from an interview with Microsoft. Given an array of random integers, write an algorithm in C that removes duplicated numbers and return the unique numbers in the original array. E.g Input: {4, 8, 4, 1, 1, 2, 9} Output: {4, 8, 1, 2, 9, ?, ?} One caveat is that the expected algorithm should not required the...

MySql "INSERT … ON DUPLICATE KEY UPDATE" still inserting duplicate records. What am I missing?

I have a simple table set up with two columns, each column is a key value. the values stored in each field are varchar(45) representing an email address and a keyword. It is possible that the information collected may duplicate itself as it is related to site browsing data collection. To avoid duplicate entries, I used tried to use INSER...

In python, how to check if there are any duplicates in list

example 1: ['one', 'two', 'one'] should return True. example 2: ['one', 'two', 'three'] should return False. ...

No duplicates in SQL query

I'm doing a select in MySQL with an inner join: SELECT DISTINCT tblcaritem.caritemid, tblcar.icarid FROM tblcaritem INNER JOIN tblprivatecar ON tblcaritem.partid = tblprivatecar.partid INNER JOIN tblcar ON tblcaritem.carid = tblcar.carid WHERE tblcaritem.userid=72; Sometimes I get duplicates of tblcaritem.caritemid in the result. ...

dealing with duplicates in a bst

My bst has to be able to cope with duplicate entries. Does anyone have any strategies for how to go about this that doesn't require excessive amounts of code? I thought of consistently adding duplicates to the right but then that will mess up the bst order. for example what happens when the duplicate has two children who in turn have two...

GROUP_CONCAT and DISTINCT are great, but how do i get rid of these duplicates i still have?

hi, i have a mysql table set up like so: id uid keywords -- --- --- 1 20 corporate 2 20 corporate,business,strategy 3 20 corporate,bowser 4 20 flowers 5 20 battleship,corporate,dungeon what i WANT my output to look like is: 20 corporate,business,strategy,bowser,flowers,battleship,dungeon b...

How to check for a duplicate email address in PHP, considering Gmail ([email protected])

How can I check for duplicate email addresses in PHP, with the possibility of Gmail's automated labeler and punctuation in mind? For example, I want these addressed to be detected as duplicates: [email protected] [email protected] [email protected] [email protected] Despite what Daniel A. White...

How do I prevent the loading of duplicate rows in to an Oracle table?

I have some large tables (millions of rows). I constantly receive files containing new rows to add in to those tables - up to 50 million rows per day. Around 0.1% of the rows I receive are duplicates of rows I have already loaded (or are duplicates within the files). I would like to prevent those rows being loaded in to the table. I ...

Finding duplicates in an NSMutableArray

I have a class (colorClass) which holds 2 NSStrings (idNumber and favoriteColor). There is an NSMutableArray (arrayColor) that holds over 50,000 colorClass objects. What is the fastest way to find all duplicate idNumbers from all colorClass objects and return them in an array? Right now I'm using 1 for loop that goes copies arrayColor...

Fastest/One-liner way to remove duplicates (by key) in Ruby Array?

What's the fastest/one-liner way to remove duplicates in an array of objects, based on a specific key:value, or a result returned from a method? For instance, I have 20 XML Element nodes that are all the same name, but they have different "text" values, some of which are duplicates. I would like to remove the duplicates by saying "if e...

How to find out if there are any duplicates in one list versus another?

I have a List where MyClass has a property 'Name'. I want to know if there are duplicate MyClass with the same Name in the list. Also, I have a different List and I want to know if there are any duplicates compared to List A. ...

prevent duplicate invoice no. against particular ID

I have confusion on how to prevent duplicate InvoiceNo against CompanyId. I just have prepare an Invoice project that has field like CompanyId and InvNo. Both do not have Unique Keys because Company ID and InvoiceNo both have to repeated. as per below CompanyID InvNo 1 1 2 1 1 2 3 1 4 ...

How to remove duplicate paragraphs in multiple files?

I have two sets of files with newspaper articles in them; about 20 files with about 2000 articles, and 1 file with about 100 articles. The 100 articles in the single file should be disjoint from the others, but are in fact duplicated randomly throughout the 20 files (once each). Any ideas for an easy way to find and remove the 100 dupl...

Deleting Duplicates in Access 2003

I have an Access 2003 table with ~4000 records which was made from 17 different tables. Roughly half of these records are duplicates. There is no unique identifying column (id, name etc). There is an id column which was auto filled when the tables were combined meaning that the duplicates aren't completely identical (though this column c...

Show duplicates in Mathematica

In Mathematica I have a list: x = {1,2,3,3,4,5,5,6} How will I make a list with the duplicates? Like: {3,5} I have been looking at Lists as Sets, if there is something like Except[] for lists, so I could do: unique = Union[x] duplicates = MyExcept[x,unique] (Of course, if the x would have more than two duplicates - say, {1,2,2,2...