curl

Using cURL with PHP, cannot get headers for a minority of sites

Hi, I'm having trouble using curl to retrieve headers for a minority of sites. Some examples are digg.com and microsoft.com. function get_headers_curl($url, $port) { $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_HEADER, true); curl_setopt($ch, CURLOPT_NOBODY, ...

Change URL location on CURL requests?

Hi, I have a page that is making an XML POST-Request to an URL on a different server (e.g. "http://wwww.externalserver.com/login"). This server is sending plain HTML as response. The problem is, that the URL shown in the browser ist still on my local server, let's say "http://localhost/callExternal.php". The external URL is redirecting ...

Bash script for downloading files with Curl

I'm trying to knock together a little batch script for downloading files, that takes a URL as its first parameter and a local filename as its second parameter. In testing I've learned that its tripping up on spaces in the output filename, so I've tried using sed to escape them, but its not working. #!/bin/bash clear echo Downloading $1...

How do i post multipart/form-data that contain symbol

It made me have a headache for hours, i try to post image to twitpic with my twitter account using curl. curl -F "username=myusername" -F "password=='hwsh[g" -F "message=test twitpic" -F media=@/dos/smallapps2/smallapps/tiny-delicious/snapshot/test.png http://twitpic.com/api/uploadAndPost It alway return <?xml version="1.0" enco...

PHP curl posting to aspx page causing 500 server error

Hello, This is a little complex question and a little hard to explain. We have build a script to auto login and collect the information from one asp.net based website, the script is in PHP using curl. We have been testing the script and it works 20-30% of the time which means the script is complete and works. But rest of the time it fa...

Parsing curl response

i'm using curl to retrieve a page which has a table structure as below. ... <tr> <td> <table>table1</table> <table>table2</table> .................... <table>table25</table> </td> </tr> .... i need the data in table table1 to table 25. How to parse it to ...

Convert Curl retrieved HTML string to JSON with PHP, serve with AJAX

Hi, I am building a scraper with Javascript (AJAX; Prototype) and PHP (Curl). The url is served trough AJAX to the PHP/Curl. The response is a huge HTML string. I would like to send the string in JSON to Javascript so I can process it. If I send the raw responseText it works just fine, the html (string) get's rendered on my screen. Ho...

how to set up curl in php?

i am trying to use some sms gateway and it require me to use curl to post a message to their server. How do i set up curl on windows and linux server? I guess curl is some php library? Which files do i need to include? Where will i get those files? ...

Need help adding a bookmark to Delicious using cURL

I'm trying to write a class that can add a bookmark to my Delicious account. Here's my method: public function addBookmark($url, $description) { $dusername = 'myUsername'; $dpassword = 'myPassword'; $api = 'api.del.icio.us/v1'; $link = urlencode($url); $desc = urlencode($description); $apicall = "https://$dusername:$d...

How do I pass cookies on a CURL redirect?

Hi, imagine the following scenario: I open a CURL connection and pass some XML-Logindata via POST. The server answers with an 302 redirect, where the session cookies are set and redirects me to a following "welcome"-page. If I enable FOLLOWLOCATION the cookies set on the redirection-page get lost and the welcome-page fails with a "sessi...

PHP Authenticate Playstation Network

Summary of what im trying to accomplish: I play motorstorm on PS3 and for the first game Monument Valley, the leaderboards are not available on the ms website. which means i cant scrape leaderboard data. Now i wiresharked my ps3 and i saw the request that monument valley uses to get the leaderboard data. now i replicated the request head...

400 Bad Request when getting a page with PHP Curl

I keep getting a 400 bad request code from a hotfile.com page when I try to get it with curl. I can get the pages fine in the browser The first (login) (post) request in the script works The last (get) request in the for loop works I have tried setting the curl headers to the same headers sent by my browser I have tried sleeping up to ...

HTTPS Form Submission with cURL(or anything better)

function login_board($url,$ch) { $cookie="cookie.txt"; curl_setopt( $ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1" ); curl_setopt( $ch, CURLOPT_URL, $url ); curl_setopt( $ch, CURLOPT_COOKIEJAR, $cookie ); curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true ); ...

i have a curl that works in localhost but not in live website. i cannot figure out why.

i have this link http://www.bata.com.sg, this website actually exists that works in my curl code that checks if the page exists. it works in my localhost code, but it keeps failing in my live website. I have tested using other domains like http://www.yahoo.com.sg, it works all the time on my localhost AND my live website. i copied t...

Determine the file size before dowloading data through HTTP

Is this possible to do? I'm currently coding in PHP using cURL library but this rather applies to HTTP as a whole. The most obvious way sounded like throwing a HEAD request to the data URL and read its Content-Length header, but the problem is that some servers including apache 2.0 does not send Content-Length against HEAD requests and ...

Joomla site on LAMP server behind proxy cannot access HTTP resource stream

Setup: 1. Joomla 1.5 website on a LAMP server (CentOS 5.2/Apache 2.2/PHP 5.2.9/mysql 5) 2. Joomla module for currency conversion added. Module uses google finance to convert currency 3. LAMP stack resides in the intranet behind a proxy. The server environment variables for http_proxy, yum.conf proxy have been setup, and kernel successful...

What is the term for a server looking up domains it hosts?

I am dealing with a server on which curl/wget/links/lynx cannot load any sites which are hosted on the server itself. External sites can be loaded/fetched etc just fine. I want to know the terms I should be using to express this to the sysadmins who insist there is no problem. Thanks! David ...

Is CURL expensive compared file_get_contents()?

Im running a video link aggregator, and I have a script that checks to see if the video was deleted from site. Its done by getting the HTML output of the link, and checking against target keywords. Currently I use file_get_contents() to get the html code of the link. The problem is, some sites redirect to another URL if the link is remo...

Post to twitter using oauth with from

I am unclear on how to post to twitter using oauth. Please do not just post a link on this page for me to look at because chances are I have already seen it. I have the section where they go to twitter and approve the app and come back to my site. I then store the information in access key and access secret key in the database from the u...

Using PHP, how do you store credentials in a cookie with basic authentication?

This seems simple, but I am stuck. I am trying to use basic authentication using an API. I can use curl from the command line, but I can't figure out how to convert this to PHP and cookies. Please help! ...