Categories:

Site Search:

Find me at:


Share/Save/Bookmark

Use PHP and cURL to retrieve URL contents (with a few options)

url_get_contents - a function to retrieve the contents of a remote URL with PHP and the cURL library.Because I've used this function for a few different projects, it's grown to be a bit of a monster, however you can still use it in a pretty basic way, i.e. url_get_contents("http://www.example.com");

I'm not a professional programmer by any means, and I suffer from lack of organisation of common code that I use. So, I'm going to post a few of the things that I hunt for here, to save myself a bit of a search. Hopefully they may be useful to someone else too.

Anyway, here's the function, with comments that should hopefully explain the logic.

<?php

# url_get_contents function by Andy Langton: http://andylangton.co.uk/
function url_get_contents($url,$useragent='cURL',$headers=false,
$follow_redirects=false,$debug=false) {

# initialise the CURL library
$ch = curl_init();

# specify the URL to be retrieved
curl_setopt($ch, CURLOPT_URL,$url);

# we want to get the contents of the URL and store it in a variable
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);

# specify the useragent: this is a required courtesy to site owners
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);

# ignore SSL errors
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

# return headers as requested
if ($headers==true){
curl_setopt($ch, CURLOPT_HEADER,1);
}

# only return headers
if ($headers=='headers only') {
curl_setopt($ch, CURLOPT_NOBODY ,1);
}

# follow redirects - note this is disabled by default in most PHP installs from 4.4.4 up
if ($follow_redirects==true) {
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); 
}

# if debugging, return an array with CURL's debug info and the URL contents
if ($debug==true) {
$result['contents']=curl_exec($ch);
$result['info']=curl_getinfo($ch);
}

# otherwise just return the contents as a variable
else $result=curl_exec($ch);

# free resources
curl_close($ch);

# send back the data
return $result;
}

?>

03.06.2008. 22:17

answerg said on 29.10.2008. 13:49

thanks a lot.

Eligio said on 24.03.2009. 20:52

Is there a way to use this function to duplicate a site then used owned url instead on the original?

Andy said on 24.03.2009. 22:08

You mean like a mirror or web proxy? It's certainly possible to use curl to do that, but might be stretching it a bit.

I don't think it would be too difficult to do (the function above wouldn't necessarily help though).

There are pre-made scripts for that sort of thing anyway. I found the below, for instance, searching Google for [php web proxy]:

http://sourceforge.net/projects/php-proxy/

JJ said on 10.07.2009. 15:39

What eligio wants is to clone/rip off/copy an existing site so he doesn't have to build his own.

Cheap scammer. Internet garbage.

Questions, comments, insults or praise? Have your say:

:

:

:


5 + 1 =

Articles RSS feed

Page last (manually) updated: December 23, 2009.