ansaurus

Question

How can I gzip standard in to a file and also print standard in to standard out?

Answer 1

+23 A:

echo "hey hey, we're the monkees" | tee /dev/tty | gzip --stdout > my_log.gz

As pointed out in the comments, /dev/stdout might work better than /dev/tty in some circumstances.

Paul Tomblin 2009-02-20 19:51:17

_A_w_e_s_o_m_e_. Thank you.

Ross Rogers 2009-02-20 19:54:29

What's /dev/tty doing there? The original question wanted the output on standard output, not necessarily on the terminal.

Gareth Rees 2009-02-20 20:29:46

/dev/tty is a synonym for the current terminal. The questioner used "standard out" in the customary manner to mean the the current terminal, rather than a more strict definition of the term.

Paul Tomblin 2009-02-20 20:41:57

If it were indeed customary to use "standard output" to mean "current terminal", then that would be a custom likely to lead to much confusion! For this question, bash has /dev/stdout.

Gareth Rees 2009-02-20 21:11:27

Under what circumstances would /dev/tty not produce the desired outcome where /dev/stdout would?

Paul Tomblin 2009-02-20 21:13:43

When stdout goes to a file and not to the terminal.

Gareth Rees 2009-02-26 13:10:23

This is just plain not what was asked -- what was asked for was a shell pipeline that would send the output to its `stdout` AND gzip it to a file. (What if I wanted to send the output on to yet another program -- say, **less(1)**.

SamB 2010-04-27 00:23:37

@SamB, what part of your ass did you pull that requirement out of? The person who asked the question was satisfied with my answer.

Paul Tomblin 2010-04-27 01:00:59

Answer 2

+12 A:

Have a nice cup of tee!

The tee command copies standard input to standard output and also to any files given as arguments. This is useful when you want not only to send some data down a pipe, but also to save a copy

As I'm having a slow afternoon, here's some gloriously illustrative ascii-art...

           +-----+                   +---+                  +-----+  
stdin ->   |cmd 1|    -> stdout ->   |tee|   ->  stdout  -> |cmd 2|
           +-----+                   +---+                  +-----+
                                       |
                                       v
                                     file

As greyfade demonstrates in another answer the 'file' need not be a regular file, but could be FIFO letting you pipe that tee'd output into a third command.

           +-----+                   +---+                  +-----+  
stdin ->   |cmd 1|    -> stdout ->   |tee|   ->  stdout  -> |cmd 2|
           +-----+                   +---+                  +-----+
                                       |
                                       v
                                     FIFO
                                       |
                                       v
                                    +-----+
                                    |cmd 3|
                                    +-----+

Paul Dixon 2009-02-20 19:53:38

But I want to gzip the intermediary file on the fly. Is that possible only using tee?

Ross Rogers 2009-02-20 19:55:31

yes, the other paul wrote a nice succinct answer while I messed about with an ascii art diagram :)

Paul Dixon 2009-02-20 19:59:23

But +1 for the excellent explanation.

Paul Tomblin 2009-02-20 20:00:36

:-PI'm familiar with tee'ing to a file like you're describing. I just don't know how to gzip that file. if i do "echo foo | tee bar.log", I don't know how to make tee gzip 'bar.log', aside from the solution Paul Tomblin posted.

Ross Rogers 2009-02-20 20:01:31

+1 for the ASCII art.

Chris Lutz 2009-02-20 20:01:53

man, stackoverflow is awesome. I didn't even have to harass the office linux guru.Thanks all.

Ross Rogers 2009-02-20 20:02:27

greyfade's excellent answer shows how you can do what you like with the tee'd output.

Paul Dixon 2009-02-20 20:16:50

Yes, grayfade's answer does precisely what this one suggests, but with a very succinct syntax. This ones `cmd 3` is written `>(cmd 3)` there.

SamB 2010-04-27 00:30:03

Answer 3

+17 A:

Another way (assuming a shell like bash or zsh):

echo "hey hey, we're the monkees" | tee >(gzip --stdout > my_log.gz)

The admittedly strange >() syntax basically does the following:

Create new FIFO (usually something in /tmp/)
Execute command inside () and bind the FIFO to stdin on that subcommand
Return FIFO filename to command line.

What tee ends up seeing, then, is something like:

tee /tmp/arjhaiX4

All gzip sees is its standard input.

For Bash, see man bash for details. It's in the section on redirection. For Zsh, see man zshexpn under the heading "Process Substitution."

As far as I can tell, the Korn Shell, variants of the classic Bourne Shell (including ash and dash), and the C Shell don't support this syntax.

greyfade 2009-02-20 20:05:32

Cool, I'm learning something myself here. Can you elaborate on what is happening there?

Paul Dixon 2009-02-20 20:08:34

I'll edit my post.

greyfade 2009-02-20 20:09:55

if I read it right, instead of given a file to tee, you've got it sending the copy as input to bracketed expression which writes the gzip output to another file. The uncompressed data leaves tee as normal on stdout

Paul Dixon 2009-02-20 20:14:04

thanks for the edit, appreciate the extra info! Never knew about that FIFO syntax.

Paul Dixon 2009-02-20 20:15:35

It came as a shock to me when I first learned it. But it lets you get away with very complex redirections, with several programs operating on the same input simultaneously. Of course, you can also do `mkfifo` by hand and run all of these commands in different consoles if you need to.

greyfade 2009-02-20 20:17:42

It also works the other way with <()

greyfade 2009-02-20 20:18:25

I've been doing Unix for 20 years, and I haven't wrapped my head around this new syntax yet.

Paul Tomblin 2009-02-20 20:23:05

Answer 4

+2 A:

Just to post a way that doesn't involve touching disk:

echo "hey hey, we're the monkees" | (exec 1>&3 && tee /proc/self/fd/3 | gzip --stdout > my_log.gz)

Joshua 2010-01-04 22:52:09

What, you mean /tmp is on a disk ?!?!?!?

SamB 2010-04-27 00:30:19

From a security perspective, it is.

Joshua 2010-04-27 00:36:44

ansaurus

tags:

views:

answers:

How can I gzip standard in to a file and also print standard in to standard out?

related questions