I made a program for serializing novels a few days ago. Because I was afraid of the trouble of updating, I wrote a collector by the way to collect information from the Eight Route Chinese Network. The function is relatively simple and the rules cannot be customized, but the general idea is in it. Custom rules can be extended by yourself.
Using PHP as a collector mainly uses two functions: file_get_contents() and preg_match_all(). The former is for remotely reading web page content, but it can only be used in versions above PHP5. The latter is a regular function. Used to extract the required content.
Now let’s talk about the function implementation step by step.
Since we are collecting novels, we must first extract the book title, author, and genre. Other information can be extracted as needed.
The goal here is "Returning to the Ming Dynasty to Become a Prince", first open the bibliography page, link: http://www.webjx.com/Book/3727/Index.aspx
Open a few more books and you will find that the basic format of the book title is: http://www.webjx.com/Book/ISBN/Index.aspx, so we can make a start page and define an , used to enter the ISBN that needs to be collected. In the future, the ISBN that needs to be collected can be received in the format of $_POST['number']. After receiving the book number, the next thing to do is to construct the bibliography page: $url=http://www.86zw.com/Book/$_POST['number']/Index.aspx. Of course, this is an example, mainly for It is easy to explain. It is best to check the legality of $_POST['number'] when actually making it.
After constructing the URL, you can start collecting book information. Use the file_get_contents() function to open the bibliographic page: $content=file_get_contents($url), so that all the contents of the bibliographic page can be read out. The next step is to match information such as book title, author, and genre. Here we take the title of the book as an example, everything else is the same. Open the bibliography page, view the source file, and find ""Back to the Ming Dynasty to Be a Prince"". This is the title of the book to be extracted. Regular expression to extract the book title: /(.*?)/is, use the preg_match_all() function to extract the book title: preg_match_all("/(.*?)/is",$contents,$title); In this way, the content of $title[0][0] is the title we want (the usage of preg_match_all function can Go to Baidu to check, I won’t explain it in detail here). After fetching the book information, the next step is to fetch the chapter content. To fetch the chapter content, the first thing to do is to find the address of each chapter, then open the chapter remotely, use regular rules to fetch the content, and store it in the database or directly generate an html static file. . This is the address of the chapter list: http://www.webjx.com/Html/Book/18/3727/List.shtm. It can be seen that this is the same as the bibliography page, and there are rules to be found: http://www .webjx.com/Html/Book/category number/book number/List.shtm. The ISBN has been obtained previously. The key here is to find the classification number. The classification number can be found on the previous bibliography page. Extract the classification number:
preg_match_all("/Html/Book/[0-9]{1,}/[0-9]{1,}/List.shtm/is",$contents,$typeid);This is not enough. A cutting function is required:
PHP code is as follows:
function cut($string,$start,$end){
$message = explode($start,$string);
$message = explode($end,$message[1]); return $message[0];} where $string is the content to be cut, $start is the starting point, and $end is the ending point. Take out the classification number:
$start = "Html/Book/";
$end
= "List.shtm";
$typeid = cut($typeid[0][0],$start,$end );
$typeid = explode("/",$typeid);[/php]
In this way, $typeid[0] is the classification number we are looking for. The next step is to construct the address of the chapter list: $chapterurl = http://www.webjx.com/Html/Book/.$typeid[0]/$_POST[‘number’]/List.shtm. With this you can find the address of each chapter. Here’s how:
$ustart = """;
$uend
= """;
//t represents the abbreviation of title
$tstart = ">";
$tend
= "<";
//Get the path, for example: 123.shtm,2342.shtm,233.shtm
preg_match_all("/"[0-9]{1,}.(shtm) "/is",$chapterurl,$url);
//Get the title, for example: Chapter 1 The Nine Worlds of Good People
preg_match_all("//is",$file,$title);
$count = count($url[0]);
for($i=0;$ i<=$count;$i++)
{
$u = cut($url[0][$i],$ustart,$uend);
$t = cut($title[0 ][$i],$tstart,$tend);
$array[$u] = $t;
}
The $array array is all the chapter addresses. At this point, the collector is half done. The rest is to loop through each chapter address, read it, and then match the content. This is relatively simple and will not be described in detail here. Okay, let’s end it today. This is my first time writing such a long article, so there will inevitably be problems with language organization. Please bear with me!
