How to grab live text from a URL?

Question

Im trying to grab all data(text) coming from a URL which is constantly sending text, I tried using PHP but that would mean having the script running the whole time which it isn’t really made for (I think). So I ended up using a BASH script.

At the moment I use wget (I couldn’t get CURL to output the text to a file)

wget --tries=0 --retry-connrefused http://URL/ --output-document=./output.txt

So wget seems to be working pretty well, apart from one thing, every time I re-start the script wget will clear the output.txt file and start filling it again, which isn’t what I want. Is there a way to tell wget to append to the txt file?

Also, is this the best way to capture the live stream of data? Should I use a different language like Python or …?

Answer 1

You could try this:

while true
do
wget -q -O - http://example.com >> filename # -O - outputs to the screen
sleep 2 # sleep 2 sec
done

Answer 2

curl http://URL/ >> output.txt

the >> redirects the output from curl to output.txt, appending to any data already there. (If it was just > output.txt - that would overwrite the contents of output.txt each time you ran it).

Answer 3

You can do wget --tries=0 --retry-connrefused $URL -O - >> output.txt.

Explanation: the parameters -O is short for --output-document, and a dash - means standard output.

The line command > file means write "write output of command to file", and command >> file means "append output of command to file" which is what you want.

Answer 4

Curl doesn't follow redirects by default and outputs nothing if there is a redirect. I always specify the --location option just in case. If you want to use curl, try:

curl http://example.com --location --silent >> output.txt

The --silent option turns off the progress indicator.