BASH MySQL Query to Comma Separated File

Question

I have the following bash script:

#!/bin/sh

MYSQLHOST="mysql.remote-host.com"
MYSQLDB="mysqldb"
MYSQLTABLE="table"
MYSQLUSER="user"
MYSQLPASS="pass"
MYSQLDUMP="Report/report.csv"
LOG="Report/report.log"

MYSQLOPTS="--user=${MYSQLUSER} --password=${MYSQLPASS} --host=${MYSQLHOST} ${MYSQLDB}"


echo "Report Begin: $(date)"
 echo "MySQL Dump Begin: $(date)" >> ${LOG}
  echo "SELECT ${MYSQLTABLE}.created_at, ${MYSQLTABLE}.product_options FROM ${MYSQLTABLE} WHERE ${MYSQLTABLE}.product_id=1 ORDER BY ${MYSQLTABLE}.created_at" | mysql ${MYSQLOPTS} > ${MYSQLDUMP}
 echo "MySQL Dump End: $(date)" >> ${LOG}
echo "Report Successful: $(date)"

This ouputs my MySQL Query into a "TAB" separated file report.csv. However, I need to have it output to a "COMMA" separated file. I realize I could create another script to convert this file from TAB to COMMA separated, however, I'd rather save the step if I can. So how can I have MySQL dump the file in comma separated format?

EDIT: I did find this solution: http://stackoverflow.com/questions/707473/how-do-you-output-mysql-query-results-in-csv-format-to-the-screen-not-to-a-file

However I can't seem to get it to work:

echo "SELECT CONCAT_WS(',', ${MYSQLTABLE}.created_at, ${MYSQLTABLE}.product_options) FROM ${MYSQLTABLE} WHERE ${MYSQLTABLE}.product_id=1 ORDER BY ${MYSQLTABLE}.created_at" | mysql ${MYSQLOPTS} > ${MYSQLDUMP}

Doesn't work :(

Answer 1

Here's an example SQL query that outputs to a comma-separated file:

SELECT a,b,a+b INTO OUTFILE '/tmp/result.txt'
  FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
  LINES TERMINATED BY '\n'
  FROM test_table;

---

Re comments: Fair enough, the INTO OUTFILE syntax writes to a file on the database server host, not the host where the shell script and the mysql client is running.

The mysql client doesn't have a csv output mode, it only has the bars-and-dashes tabular format (with the --table or -t flags), tab-separated format (with the --batch or -B flags), HTML format (--html or -H) and XML (--xml or -X).

If you have content in your database that contains tabs, then either convert tabs to something else using the REPLACE() SQL function, or else output in one of the other formats and try to convert that to csv:

mysql ... --table --skip-column-names |
sed -e 's/  *|  */,/g' \
    -e 's/^|  *//' \
    -e 's/  *|$//' \
    -e '/^+--/d'

Answer 2

You don't need another script to do that. Just change:

> ${MYSQLDUMP}

at the end of your line to read:

| sed 's/\x09/,/g' > ${MYSQLDUMP}

But you need to be careful that your data doesn't have commas in it or they'll be mis-recognized by whatever uses the CSV file (but you may have that problem with tabs already).

Answer 3

Why not use

SELECT ${MYSQLTABLE}.created_at, ${MYSQLTABLE}.product_options INTO outfile '${MYSQLDUMP}' FROM ${MYSQLTABLE} FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' WHERE ${MYSQLTABLE}.product_id=1 ORDER BY ${MYSQLTABLE}.created_at

Answer 4

Found a simple perl 1-liner that seems to do the trick:

perl -F"\t" -lane 'print join ",", map {s/"/""/g; /^[\d.]+$/ ? $_ : qq("$_")} @F '

-Perl Script

So the line in my script looks like this:

     echo "SELECT ${MYSQLTABLE}.created_at, ${MYSQLTABLE}.product_options FROM ${MYSQLTABLE} WHERE ${MYSQLTABLE}.product_id=1 ORDER BY ${MYSQLTABLE}.created_at" | mysql ${MYSQLOPTS} | perl -F"\t" -lane 'print join ",", map {s/"/""/g; /^[\d.]+$/ ? $_ : qq("$_")} @F ' > ${MYSQLDUMP}

Thanks for all the help, Especially from Bill Karwin, I wanted to add a VoteUp to your answer but I don't have enough reputation :( Once I get more rep, I'll circle back and vote your answer up. Thanks again.

Answer 5

Run the following:

echo "SELECT CONCAT_WS(',', ${MYSQLTABLE}.created_at, ${MYSQLTABLE}.product_options) FROM ${MYSQLTABLE} WHERE ${MYSQLTABLE}.product_id=1 ORDER BY ${MYSQLTABLE}.created_at" | \
mysql ${MYSQLOPTS}  | tr '\t' ',' > ${MYSQLDUMP}

The tr command replaces one char with another.