tags:

views:

35

answers:

1

I have the following query in mysql 5.1.41:

select distinct table_schema from information_schema.tables where table_schema like '%dog%';

I want to take the output of that command:

+-------------------+
| table_schema      |
+-------------------+
| dog_a             |
| dog_b             |
+-------------------+

and then use the database names as input to a query like the following:

select count(*) from (select * from dog_a.log where insane = 1 UNION ALL select * from dog_b.log where insane = 1) as total_count;

such that the algorithm is essentially:

For each database in databases, count the number of insane dogs and sum the total across all databases. However, I don't know how to wrap up the two queries to get the database names from the first query as iterable input into the second query within mysql.

I need to be able to do this entirely within the database.

Any ideas?

Thanks!

+1  A: 

Something like this might do. I haven't got a bunch of 'dogs-with-logs' to test this against, but I tried a slightly edited version, and the basic idea seems to work.

Build up the query string in a variable, then use a prepared statement to execute it.

SELECT @query:=CONCAT(
      'select count(*) from ('
    , GROUP_CONCAT( CONCAT( y.prefix, x.table_schema, y.postfix ) SEPARATOR ' UNION ALL ' )
    , ') as total_count' )
FROM (
    SELECT  DISTINCT table_schema
    FROM    information_schema.tables
    WHERE   table_schema LIKE '%dog%'
    ) AS x
JOIN (
    SELECT
          'select * from '        AS prefix
        , '.log where insane = 1' AS postfix 
    ) AS y
;

-- SELECT @query AS Query;

PREPARE STMT FROM @query;
EXECUTE STMT;
DEALLOCATE PREPARE STMT;
martin clayton
This solves my problem on mysql 5.1. However, I noticed it breaks on mysql 5.0. The problem seems to be that the generated intermediate union query is truncated at about 10% of the tables into the query. The truncation is clear because the select statement is cut in half. I'll pursue that problem, but if you know what might be causing that, I'm all ears. It looks like a bug to me. This is running mysql 5.0.77 on centos. Thank you.
David Watson
I figured out what the problem was with mysql 5.0 on centos. The GROUP_CONCAT_MAX_LEN defaults to 1024 which is way too short depending on how many databases you have defined. In this case we have ~100 so the generated SQL got truncated in the GROUP_CONCAT. The solution is to set the GROUP_CONCAT_MAX_LEN to something large enough in the session such as: set session group_concat_max_len=102400;
David Watson
@David - you had a higher value set on 5.1 then?
martin clayton
@martin - no, I had few enough databases on my development machine that I didn't go over the 1024 limit. That was the difference between my ubuntu dev machine running mysql 5.1 and the centos production machine running mysql 5.0. Consistency is a wonderful thing. :)
David Watson
@martin: Thanks so much for your help. This was a quantum leap for us in terms of being able to do these queries generically across databases. This will form the basis of a lot of interesting data analysis and visualization tools for us.
David Watson
@David - sounds exciting - thanks for your thanks!
martin clayton