Web Scraping with PHP

50 %
50 %
Information about Web Scraping with PHP
Technology

Published on September 20, 2008

Author: tobias382

Source: slideshare.net

Web Scraping with PHP Matthew Turland September 16, 2008

Everyone acquainted? Lead Programmer for surgiSYS, LLC PHP Community member Blog: http://ishouldbecoding.com

Lead Programmer for surgiSYS, LLC

PHP Community member

Blog: http://ishouldbecoding.com

What is Web Scraping? 2 Stage Process Stage 1 : Retrieval GET /some/resource ... HTTP/1.1 200 OK ... Resource with data you want Stage 2 : Analysis Raw resource Usable data

How is it different from... Data Mining Focus in data mining Focus in web scraping Consuming Web Services Web service data formats Web scraping data formats

Potential Applications What Data source When Web service is unavailable or data access is one-time only. Crawlers and indexers Remote data search offers no capabilities for search or data source integration. Integration testing Applications must be tested by simulating client behavior and ensuring responses are consistent with requirements.

Disadvantages Δ == vs

Legal Concerns TOS TOU EUA Original source Illegal syndicate IANAL!

Retrieval GET /some/resource ... sizeof( ) == sizeof( ) if( ) require ;

The Low-Down on HTTP

Know enough HTTP to... Use one like this: To do this:

Know enough HTTP to... PEAR::HTTP_Client pecl_http Zend_Http_Client Learn to use and troubleshoot one like this: Or roll your own! cURL Filesystem + Streams

GET /wiki/Main_Page HTTP/1.1 Host: en.wikipedia.org Let's GET Started method or operation URI address for the desired resource protocol version in use by the client header name header value request line header more headers follow...

GET /wiki/Main_Page HTTP/1.1

Host: en.wikipedia.org

Warning about GET In principle: "Let's do this by the book." GET In reality: "' Safe operation '? Whatever." GET

URI vs URL 1. Uniquely identifies a resource 2. Indicates how to locate a resource 3. Does both and is thus human-usable. URI URL More info in RFC 3986 Sections 1.1.3 and 1.2.2

Query Strings http://en.wikipedia.org/w/index.php? title=Query_string&action=edit URL Query String Question mark to separate the resource address and query string Equal signs to separate parameter names and respective values Ampersands to separate parameter name-value pairs. Parameter Value

URL Encoding Parameter Value first second this is a field was it clear enough (already)? Query String first=this+is+a+field&second=was+it+clear+%28already%29%3F Also called percent encoding . urlencode and urlencode : Handy PHP URL functions $_SERVER ['QUERY_STRING'] / http_build_query ( $_GET ) More info on URL encoding in RFC 3986 Section 2.1

POST Requests Most Common HTTP Operations 1. GET 2. POST ... /w/index.php POST /new/resource -or- /updated/resource GET /some/resource HTTP/1.1 Header: Value ... POST /some/resource HTTP/1.1 Header: Value request body none

POST Request Example POST /w/index.php?title=Wikipedia:Sandbox HTTP/1.1 Content-Type: application/x-www-form-urlencoded wpStarttime=20080719022313&wpEdittime=20080719022100... Blank line separates request headers and body Content type for data submitted via HTML form (multipart/form-data for file uploads ) Request body ... look familiar? Note : Most browsers have a query string length limit. Lowest known common denominator: IE7 – strlen(entire URL) <= 2,048 bytes. This limit is not standardized and only applies to query strings, not request bodies.

POST /w/index.php?title=Wikipedia:Sandbox HTTP/1.1

Content-Type: application/x-www-form-urlencoded

wpStarttime=20080719022313&wpEdittime=20080719022100...

HEAD Request HEAD /wiki/Main_Page HTTP/1.1 Host: en.wikipedia.org Same as GET with two exceptions: 1 HTTP/1.1 200 OK Header: Value 2 No response body HEAD vs GET Sometimes headers are all you want ?

HEAD /wiki/Main_Page HTTP/1.1 Host: en.wikipedia.org

HTTP/1.1 200 OK Header: Value

Responses HTTP/1.0 200 OK Server: Apache X-Powered-By: PHP/5.2.5 ... [body] Lowest protocol version required to process the response Response status code Response status description Status line Same header format as requests, but different headers are used (see RFC 2616 Section 14 )

Response Status Codes 1xx Informational Request received, continuing process. 2xx Success Request received, understood, and accepted. 3xx Redirection Client must take additional action to complete the request. 4xx Client Error Request is malformed or could not be fulfilled. 5xx Server Error Request was valid, but the server failed to process it. See RFC 2616 Section 10 for more info.

1xx Informational Request received, continuing process.

2xx Success Request received, understood, and accepted.

3xx Redirection Client must take additional action to complete the request.

4xx Client Error Request is malformed or could not be fulfilled.

5xx Server Error Request was valid, but the server failed to process it.

Headers Set-Cookie Cookie Location Watch out for infinite loops! Last-Modified If-Modified-Since 304 Not Modified ETag If-None-Match OR See RFC 2109 or RFC 2965 for more info.

More Headers WWW-Authenticate Authorization User-Agent 200 OK / 403 Forbidden See RFC 2617 for more info. User-Agent: Some servers perform user agent sniffing Some clients perform user agent spoofing

Best Practices If responses are unlikely to change often, cache them locally and check for updates with HEAD requests. If possible, measure target server response time and vary when requests are sent based on peak usage times. Replicate normal user behavior (particularly headers) as closely as possible. If the application is flexible, vary behavior where it makes sense for performance.

If responses are unlikely to change often, cache them locally and check for updates with HEAD requests.

If possible, measure target server response time and vary when requests are sent based on peak usage times.

Replicate normal user behavior (particularly headers) as closely as possible. If the application is flexible, vary behavior where it makes sense for performance.

Simple Streams Examples $uri = 'http://www.example.com/some/resource'; $get = file_get_contents ($uri); $context = stream_context_create (array('http' => array( 'method' => 'POST', 'header' => 'Content-Type: ' . 'application/x-www-form-urlencoded', 'content' => http_build_query(array( 'var1' => 'value1', 'var2' => 'value2' )) ))); // Last 2 parameters here also apply to fopen() $post = file_get_contents ($uri, false, $context);

Streams Resources Language Reference > Context options and parameters HTTP context options Context parameters Appendices > List of Supported Protocols/Wrappers HTTP and HTTPS php|architect's Definitive Guide to PHP Streams (ETA late 2008 / early 2009)

Language Reference > Context options and parameters

HTTP context options

Context parameters

Appendices > List of Supported Protocols/Wrappers

HTTP and HTTPS

php|architect's Definitive Guide to PHP Streams (ETA late 2008 / early 2009)

pecl_http Examples $http = new HttpRequest ($uri); $http-> enableCookies (); $http-> setMethod (HTTP_METH_POST); // or HTTP_METH_GET $http-> addPostFields ($postData); $http-> setOptions (array( 'httpauth' => $username . ':' . $password, 'httpauthtype' => HTTP_AUTH_BASIC, ‘ useragent’ => 'PHP ' . phpversion(), 'referer' => 'http://example.com/some/referer', 'range' => array(array(1, 5), array(10, 15)) )); $response = $http-> send (); $headers = $response-> getHeaders (); $body = $response-> getBody (); See PHP Manual for more info.

PEAR::HTTP_Client Examples $cookiejar = new HTTP_Client_CookieManager (); $request = new HTTP_Request ($uri); $request-> setMethod (HTTP_REQUEST_METHOD_POST); $request-> setBasicAuth ($username, $password); $request-> addHeader ('User-Agent', $userAgent); $request-> addHeader ('Referer', $referrer); $request-> addHeader ('Range', 'bytes=2-3,5-6'); foreach ($postData as $key => $value) $request-> addPostData ($key, $value); $request-> sendRequest (); $cookiejar-> updateCookies ($request); $request = new HTTP_Request ($otheruri); $cookiejar-> passCookies ($request); $response = $request-> sendRequest (); $headers = $request->getResponseHeader(); $body = $request->getResponseBody(); See PEAR Manual and API Docs for more info.

Zend_Http_Client Examples $client = new Zend_Http_Client ($uri); $client-> setMethod (Zend_Http_Client::POST); $client-> setAuth ($username, $password); $client-> setHeaders ('User-Agent', $userAgent); $client-> setHeaders (array( 'Referer' => $referrer, 'Range' => 'bytes=2-3,5-6' ); $client-> setParameterPost ($postData); $client-> setCookieJar (); $client-> request (); $client-> setUri ($otheruri); $client-> setMethod (Zend_Http_Client::GET); $response = $client-> request (); $headers = $response-> getHeaders (); $body = $response-> getBody (); See ZF Manual for more info.

cURL Examples Fatal error: Allowed memory size of n00b bytes exhausted (tried to allocate 1337 bytes) in /this/slide.php on line 1 See PHP Manual , Context Options , or my php|architect article for more info. Just kidding. Really, the equivalent cURL code for the previous examples is so verbose that it won't fit on one slide and I don't think it's deserving of multiple slides.

HTTP Resources RFC 2616 HyperText Transfer Protocol RFC 3986 Uniform Resource Identifiers &quot;HTTP: The Definitive Guide&quot; (ISBN 1565925092) &quot;HTTP Pocket Reference: HyperText Transfer Protocol&quot; (ISBN 1565928628) &quot;HTTP Developer's Handbook&quot; (ISBN 0672324547) by Chris Shiflett Ben Ramsey's blog series on HTTP

RFC 2616 HyperText Transfer Protocol

RFC 3986 Uniform Resource Identifiers

&quot;HTTP: The Definitive Guide&quot; (ISBN 1565925092)

&quot;HTTP Pocket Reference: HyperText Transfer Protocol&quot; (ISBN 1565928628)

&quot;HTTP Developer's Handbook&quot; (ISBN 0672324547) by Chris Shiflett

Ben Ramsey's blog series on HTTP

Analysis Raw resource Usable data DOM XMLReader SimpleXML XSL tidy PCRE String functions JSON ctype XML Parser

Cleanup tidy is good for correcting markup malformations. * String functions and PCRE can be used for manual cleanup prior to using a parsing extension. DOM is generally forgiving when parsing malformed markup. It generates warnings that can be suppressed. Save a static copy of your target, use a validator on the input (ex: W3C Markup Validator ), fix validation errors manually, and write code to automatically apply fixes.

tidy is good for correcting markup malformations. *

String functions and PCRE can be used for manual cleanup prior to using a parsing extension.

DOM is generally forgiving when parsing malformed markup. It generates warnings that can be suppressed.

Save a static copy of your target, use a validator on the input (ex: W3C Markup Validator ), fix validation errors manually, and write code to automatically apply fixes.

Parsing DOM and SimpleXML are tree-based parsers that store the entire document in memory to provide full access. XMLReader is a pull-based parser that iterates over nodes in the document and is less memory-intensive. SAX is also pull-based, but uses event-based callbacks. JSON can be used to parse isolated JavaScript data. Nothing &quot;official&quot; for CSS. Find something like CSSTidy . PCRE can be used for parsing. Last resort, though.

DOM and SimpleXML are tree-based parsers that store the entire document in memory to provide full access.

XMLReader is a pull-based parser that iterates over nodes in the document and is less memory-intensive.

SAX is also pull-based, but uses event-based callbacks.

JSON can be used to parse isolated JavaScript data.

Nothing &quot;official&quot; for CSS. Find something like CSSTidy .

PCRE can be used for parsing. Last resort, though.

Validation Make as few assumptions (and as many assertions) about the target as possible. Validation provides additional sanity checks for your application. PCRE can be used to form pattern-based assertions about extracted data. ctype can be used to form primitive type-based assertions.

Make as few assumptions (and as many assertions) about the target as possible.

Validation provides additional sanity checks for your application.

PCRE can be used to form pattern-based assertions about extracted data.

ctype can be used to form primitive type-based assertions.

Transformation XSL can be used to extract data from an XML-compatible document and retrofit it to a format defined by an XSL template. To my knowledge, this capability is unfortunately unique to XML-compatible data. Use components like template engines to separate formatting of data from retrieval/analysis logic.

XSL can be used to extract data from an XML-compatible document and retrofit it to a format defined by an XSL template.

To my knowledge, this capability is unfortunately unique to XML-compatible data.

Use components like template engines to separate formatting of data from retrieval/analysis logic.

Abstraction Remain in keeping with the DRY principle. Develop components that can be reused across projects. Ex: DomQuery , Zend_Dom . Make an effort to minimize application-specific logic. This applies to both retrieval and analysis.

Remain in keeping with the DRY principle.

Develop components that can be reused across projects. Ex: DomQuery , Zend_Dom .

Make an effort to minimize application-specific logic. This applies to both retrieval and analysis.

Assertions Apply to long-term real-time web scraping applications. Affirm conditions of behavior and output of the target application. Use in the application during runtime to avoid Bad Things (tm) happening when the target application changes. Include in unit tests of the application. You are using unit tests, right?

Apply to long-term real-time web scraping applications.

Affirm conditions of behavior and output of the target application.

Use in the application during runtime to avoid Bad Things (tm) happening when the target application changes.

Include in unit tests of the application. You are using unit tests, right?

Testing Write tests on target application output stored in local files that can be run sans internet during development. If possible/feasible/appropriate, write &quot;live tests&quot; that actively test using assertions on the target application. Run live tests when the target appears to have changed (because your web scraping application breaks).

Write tests on target application output stored in local files that can be run sans internet during development.

If possible/feasible/appropriate, write &quot;live tests&quot; that actively test using assertions on the target application.

Run live tests when the target appears to have changed (because your web scraping application breaks).

Questions? No heckling... OK, maybe just a little. I will hang around afterward if you have questions, points for discussion, or just want to say hi. It's cool, I don't bite or have cooties or anything. I have business cards too. I generally blog about my experiences with web scraping and PHP at http://ishouldbecoding.com. </shameless_plug> Thanks for coming!

No heckling... OK, maybe just a little.

I will hang around afterward if you have questions, points for discussion, or just want to say hi. It's cool, I don't bite or have cooties or anything. I have business cards too.

I generally blog about my experiences with web scraping and PHP at http://ishouldbecoding.com. </shameless_plug>

Thanks for coming!

Add a comment

Related presentations

Related pages

Basic PHP Web Scraping Script Tutorial - Oooff.com

Basic PHP Web Scraping Script Tutorial. Now that we have a basic idea on what web scraping is, let's get into some very simple scripts to do it.
Read more

The Future of the Web » Easy web scraping with PHP » Web ...

Easy web scraping with PHP ... If you are writing new web scraping code, I recommend looking at using the excellent Ultimate Web Scraper Toolkit:
Read more

Web Scraping With PHP & CURL [Part 1] | Jacob Ward

Hi im trying to email you on the hire me form but its not sending….. My msg: Hi jacob, do you know gscraper? we have 1000 web properties a week now that ...
Read more

php|architect’s Guide to Web Scraping with PHP « php ...

Book Details Title php|architect's Guide to Web Scraping with PHP ISBN 978-0981034515 Pages 192 Digital Formats PDF, ePub, Mobi Author Matthew ...
Read more

Getting Started with Web Scraping in PHP - Wern Ancheta

Have you ever wanted to get a specific data from another website but there’s no API available for it? That’s where Web Scraping comes in …
Read more

screen scraping - How to implement a web scraper in PHP ...

What built-in PHP functions are useful for web scraping? What are some good resources (web or print) for getting up to speed on web scraping with PHP?
Read more

Web Scraping with PHP | Buzu's Oficial Blog

Web scraping is an interesting thing to do. There is a lot of data on the web, and there are many interesting things that can be done with it if ...
Read more

html - Web scraping in PHP - Stack Overflow

I'm looking for a way to make a small preview of another page from a URL given by the user in PHP. I'd like to retrieve only the title of the page, an ...
Read more

Instant PHP Web Scraping: ebook jetzt bei weltbild.de

eBook Shop: Instant PHP Web Scraping von Jacob Ward als Download. Jetzt eBook sicher bei Weltbild runterladen & bequem mit Ihrem Tablet oder eBook Reader ...
Read more

web scraping in php · GitHub - Create a new Gist · GitHub

Have you ever wanted to get a specific data from another website but there's no API available for it? That's where Web Scraping comes in, if the data is ...
Read more