postgresql

Dealing with errors during a copy from

I've to import a file from an external source to a postgresql table. I tried to do it with \copy from , but I keep getting errors (additional columns) in the middle of the file. Is there a way to tell postgresql to ignore lines containing errors during a "\copy from" ? Thanks ...

Importing a large dataset into a database

I'm a beginning programmer in the relevant areas to this question, so if possible, it'd be helpful to avoid assuming I know a lot already. I'm trying to import the OpenLibrary dataset into a local Postgres database. After it's imported, I plan to use it as a starting seed for a Ruby on Rails application that will include information on ...

Running query from scratch with something like exec function?

Hi, is it possible to make something similar to the following with Postgresql without using a function? pseudo sql code: select * from sometable where somecol = somevalue AND someothercol IN exec( 'select something from exclusionlist' ) My primary intention is to build up a table with predefined queries to call inside a where clause ...

Asynchronous callback - gwt

Hi, I am using gwt and postgres for my project. On the front end i have few widgets whose data i am trying to save on to tables at the back-end when i click on "save project" button(this also takes the name for the created project). In the asynchronous callback part i am setting more than one table. But it is not sending the data prope...

Unit Test For NpgsqlCommand With Rhino Mocks

My unit test keeps getting the following error: "System.InvalidOperationException: The Connection is not open." The Test [TestFixture] public class Test { [Test] public void Test1() { NpgsqlConnection connection = MockRepository.GenerateStub<NpgsqlConnection>(); // Tried to fake the open connection c...

PostgreSql XML Text search

I have a text column in a table. We store XML in this column. Now I want to search for tags and values Example data: <bank> <name>Citi Bank</name> ..... ..... /<bank> I would like to run the following query: select * from xxxx where to_tsvector('english',xml_column) @@ to_tsquery('<name>Citi Bank</name>') This works fine...

Faster way to transfer table data from linked server

After much fiddling, I've managed to install the right ODBC driver and have successfully created a linked server on SQL Server 2008, by which I can access my PostgreSQL db from SQL server. I'm copying all of the data from some of the tables in the PgSQL DB into SQL Server using merge statements that take the following form: with mbRem...

postgresql duplicate table names best practice

My company has a handful of apps that we deploy in the websites we build. Recently a very old app needed to be included along side a newer app and there was a conflict w/ a duplicate table name needed to be used by both apps. We are now in the process of updating an old app and there will be some DB updates. I'm curious what people ...

Postgres not using enough CPU during index build!

I have a Postgres instance building a GIN index. It's looking at about 200,000 rows and it's so far taken about 9 hours. Who knows how long it will take eventually. The problem is that it's using about 2% of CPU when I'd like it to use more like 90%. Is there any way to force it to speed up? ...

Moving information between databases

I'm on Postgres, and have two databases on the same machine, and I'd like to move some data from database Source to database Dest. Database Source: Table User has a primary key Table Comments has a primary key Table UserComments is a join table with two foreign keys for User and Comments Dest looks just like Source in structure, but a...

Postgresql performance question about inserting or updating many row containing binary data

Hello, I have a table A where I put many image resources with a daily frequence. Every record of table A references another table B in which there are only fixed records. My question is the following: better to clean all records in A and then inserting new images or updating only the binary column of all records. What your advice? ...

jdbc query - date ranges as parameters

Hi all, I'd like to write a single JDBC statement that can handle the equivalent of any number of NOT BETWEEN date1 AND date2 where clauses. By single query, i mean that the same SQL string will be used to create the JDBC statements and then have different parameters supplied. This is so that the underlying frameworks can efficiently...

Speeding up PostgreSQL query where data is between two dates

I have a large table (> 50m rows) which has some data with an ID and timestamp: id, timestamp, data1, ..., dataN ...with a multi-column index on (id, timestamp). I need to query the table to select all rows with a certain ID where the timestamp is between two dates, which I am currently doing using: SELECT * FROM mytable WHERE id = ...

PostgreSQL / Ruby for commercial application

I am planning a web-based commercial application with front-end RoR and back-end PostgreSQL. I've some confusion about RoR and PostgreSQL Edition to use. For RoR, I have Aptana RADRails installed. For PostgreSQL, a free variant is also available at EnterpriseDB. Previously I installed a free EnterpriseDB PostgreSQL variant and it was v...

Django ORM leaves opened connections

I'm using django ORM with Postgres. After any operations with models (e.g. simple select) in postgres appears new opened connection in IDLE state. I've tried all possible transaction manipulations, I've tried calling connection.close() manually. All useless. And sooner or later, I'm recieveing "FATAL: connection limit exceeded for...

PGError syntax problem for named_scope

Hi, I have the following named_scope which works fine in MySQL and sqlite but bombs in Postgres: course.rb named_scope :current, :conditions => ['start < ? AND end > ? ', Time.now, Time.now], :order => 'start ASC' Then I just call: Course.current I get the error: PGError: ERROR: syntax error at or near "end" LINE 1: ... W...

Postgres: Reduce varchar size and truncate

I currently have a Postgres 8.4 database that contains a varchar(10000) column. I'd like to change this into a varchar(255) and truncate any data that happens to be too long. How can I do this? ...

Django query: Count and Group BY

I have a query that I'm trying to figure the "django" way of doing it: I want to take the last 100 calls from Call. Which is easy: calls = Call.objects.all().order_by('-call_time')[:100] However the next part I can't find the way to do it via django's ORM. I want to get a list of the call_types and the number of calls each one has WITH...

Table with a lot of attributes

Hi, I'm planing to build some database project. One of the tables have a lot of attributes. My question is: What is better, to divide the the class into 2 separate tables or put all of them into one table. below is an example create table User { id, name, surname,... show_name, show_photos, ...) or create table User { id, name, sur...

Compressing large text data before storing into db?

Hello, I have application which retrieves many large log files from a system LAN. Currently I put all log files on Postgresql, the table has a column type TEXT and I don't plan any search on this text column because I use another external process which nightly retrieves all files and scans for sensitive pattern. So the column value c...