rss-bridge/lib/contents.php

462 lines
14 KiB
PHP
Raw Normal View History

<?php
2018-11-16 23:48:59 +03:00
/**
* This file is part of RSS-Bridge, a PHP project capable of generating RSS and
* Atom feeds for websites that don't have one.
*
* For the full license information, please view the UNLICENSE file distributed
* with this source code.
*
* @package Core
* @license http://unlicense.org/ UNLICENSE
* @link https://github.com/rss-bridge/rss-bridge
*/
/**
* Exception class to handle all errors, when executing getContents
*/
class GetContentsException extends \Exception {
public function __construct($details, $code = 0, Throwable $previous = null) {
$message = trim($this->getMessageHeading() . "\n$details");
$lastError = error_get_last();
if($lastError !== null)
$message .= "\nLast PHP Error: " . $lastError['message'];
parent::__construct($message, $code, $previous);
}
protected function getMessageHeading() {
return 'Could not get contents';
}
}
/**
* Exception class to handle HTTP responses with Cloudflare challenges
**/
class CloudflareChallengeException extends \Exception {
public function __construct($code = 0, Throwable $previous = null) {
$message = <<<EOD
The server responded with a Cloudflare challenge, which is not supported by RSS-Bridge!
If this error persists longer than a week, please consider opening an issue on GitHub!
EOD;
parent::__construct($message, $code, $previous);
}
}
/**
* Exception class to handle non-20x HTTP responses
**/
class UnexpectedResponseException extends \GetContentsException {
private $responseCode;
private $responseHeaders;
private $responseBody;
protected function getMessageHeading() {
return 'Unexpected response from upstream';
}
public function __construct($responseBody, $responseHeaders, $responseCode = 500, Throwable $previous = null) {
$this->responseCode = $responseCode;
$this->responseHeaders = $responseHeaders;
$this->responseBody = $responseBody;
parent::__construct('', $responseCode, $previous);
}
public function getResponseCode() {
return $this->responseCode;
}
public function getResponseHeaders() {
return $this->responseHeaders;
}
public function getResponseBody() {
return $this->responseBody;
}
}
2018-11-16 23:48:59 +03:00
/**
* Gets contents from the Internet.
*
* **Content caching** (disabled in debug mode)
*
* A copy of the received content is stored in a local cache folder `server/` at
* {@see PATH_CACHE}. The `If-Modified-Since` header is added to the request, if
* the provided URL has been cached before.
*
* When the server responds with `304 Not Modified`, the cached data is returned.
* This will improve response times and reduce bandwidth for servers that support
* the `If-Modified-Since` header.
*
* Cached files are forcefully removed after 24 hours.
*
* @link https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since
* If-Modified-Since
*
2018-11-16 23:48:59 +03:00
* @param string $url The URL.
* @param array $header (optional) A list of cURL header.
* For more information follow the links below.
* * https://php.net/manual/en/function.curl-setopt.php
* * https://curl.haxx.se/libcurl/c/CURLOPT_HTTPHEADER.html
* @param array $opts (optional) A list of cURL options as associative array in
* the format `$opts[$option] = $value;`, where `$option` is any `CURLOPT_XXX`
* option and `$value` the corresponding value.
action: Add action to check bridge connectivity (#1147) * action: Add action to check bridge connectivity It is currently not simply possible to check if the remote server for a bridge is reachable or not, which means some of the bridges might no longer work because the server is no longer on the internet. In order to find those bridges we can either check each bridge individually (which takes a lot of effort), or use an automated script to do this for us. If a server is no longer reachable it could mean that it is temporarily unavailable, or shutdown permanently. The results of this script will at least help identifying such servers. * [Connectivity] Use Bootstrap container to properly display contents * [Connectivity] Limit connectivity checks to debug mode Connectivity checks take a long time to execute and can require a lot of bandwidth. Therefore, administrators should be able to determine when and who is able to utilize this action. The best way to prevent regular users from accessing this action is by making it available in debug mode only (public servers should never run in debug mode anyway). * [Connectivity] Split implemenation into multiple files * [Connectivity] Make web page responsive to user input * [Connectivity] Make status message sticky * [Connectivity] Add icon to the status message * [contents] Add the ability for getContents to return header information * [Connectivity] Add header information to the reply Json data * [Connectivity] Add new status (blue) for redirected sites Also adds titles to status icons (Successful, Redirected, Inactive, Failed) * [Connectivity] Fix show doesn't work for inactive bridges * [Connectivity] Fix typo * [Connectivity] Catch errors in promise chains * [Connectivity] Allow search by status and update dynamically * [Connectivity] Add a progress bar * [Connectivity] Use bridge factory * [Connectivity] Import Bootstrap v4.3.1 CSS
2019-11-01 00:02:38 +03:00
* @param bool $returnHeader Returns an array of two elements 'header' and
* 'content' if enabled.
2018-11-16 23:48:59 +03:00
*
* For more information see http://php.net/manual/en/function.curl-setopt.php
* @return string|array The contents.
2018-11-16 23:48:59 +03:00
*/
action: Add action to check bridge connectivity (#1147) * action: Add action to check bridge connectivity It is currently not simply possible to check if the remote server for a bridge is reachable or not, which means some of the bridges might no longer work because the server is no longer on the internet. In order to find those bridges we can either check each bridge individually (which takes a lot of effort), or use an automated script to do this for us. If a server is no longer reachable it could mean that it is temporarily unavailable, or shutdown permanently. The results of this script will at least help identifying such servers. * [Connectivity] Use Bootstrap container to properly display contents * [Connectivity] Limit connectivity checks to debug mode Connectivity checks take a long time to execute and can require a lot of bandwidth. Therefore, administrators should be able to determine when and who is able to utilize this action. The best way to prevent regular users from accessing this action is by making it available in debug mode only (public servers should never run in debug mode anyway). * [Connectivity] Split implemenation into multiple files * [Connectivity] Make web page responsive to user input * [Connectivity] Make status message sticky * [Connectivity] Add icon to the status message * [contents] Add the ability for getContents to return header information * [Connectivity] Add header information to the reply Json data * [Connectivity] Add new status (blue) for redirected sites Also adds titles to status icons (Successful, Redirected, Inactive, Failed) * [Connectivity] Fix show doesn't work for inactive bridges * [Connectivity] Fix typo * [Connectivity] Catch errors in promise chains * [Connectivity] Allow search by status and update dynamically * [Connectivity] Add a progress bar * [Connectivity] Use bridge factory * [Connectivity] Import Bootstrap v4.3.1 CSS
2019-11-01 00:02:38 +03:00
function getContents($url, $header = array(), $opts = array(), $returnHeader = false){
Debug::log('Reading contents from "' . $url . '"');
// Initialize cache
$cacheFac = new CacheFactory();
$cacheFac->setWorkingDir(PATH_LIB_CACHES);
$cache = $cacheFac->create(Configuration::getConfig('cache', 'type'));
$cache->setScope('server');
$cache->purgeCache(86400); // 24 hours (forced)
$params = array($url);
$cache->setKey($params);
action: Add action to check bridge connectivity (#1147) * action: Add action to check bridge connectivity It is currently not simply possible to check if the remote server for a bridge is reachable or not, which means some of the bridges might no longer work because the server is no longer on the internet. In order to find those bridges we can either check each bridge individually (which takes a lot of effort), or use an automated script to do this for us. If a server is no longer reachable it could mean that it is temporarily unavailable, or shutdown permanently. The results of this script will at least help identifying such servers. * [Connectivity] Use Bootstrap container to properly display contents * [Connectivity] Limit connectivity checks to debug mode Connectivity checks take a long time to execute and can require a lot of bandwidth. Therefore, administrators should be able to determine when and who is able to utilize this action. The best way to prevent regular users from accessing this action is by making it available in debug mode only (public servers should never run in debug mode anyway). * [Connectivity] Split implemenation into multiple files * [Connectivity] Make web page responsive to user input * [Connectivity] Make status message sticky * [Connectivity] Add icon to the status message * [contents] Add the ability for getContents to return header information * [Connectivity] Add header information to the reply Json data * [Connectivity] Add new status (blue) for redirected sites Also adds titles to status icons (Successful, Redirected, Inactive, Failed) * [Connectivity] Fix show doesn't work for inactive bridges * [Connectivity] Fix typo * [Connectivity] Catch errors in promise chains * [Connectivity] Allow search by status and update dynamically * [Connectivity] Add a progress bar * [Connectivity] Use bridge factory * [Connectivity] Import Bootstrap v4.3.1 CSS
2019-11-01 00:02:38 +03:00
$retVal = array(
'header' => '',
'content' => '',
);
2022-04-10 19:54:18 +03:00
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
2022-04-10 19:54:18 +03:00
if(is_array($header) && count($header) !== 0) {
2022-04-10 19:54:18 +03:00
Debug::log('Setting headers: ' . json_encode($header));
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
2022-04-10 19:54:18 +03:00
}
2022-04-10 19:54:18 +03:00
curl_setopt($ch, CURLOPT_USERAGENT, Configuration::getConfig('http', 'useragent'));
curl_setopt($ch, CURLOPT_TIMEOUT, Configuration::getConfig('http', 'timeout'));
curl_setopt($ch, CURLOPT_ENCODING, '');
curl_setopt($ch, CURLOPT_PROTOCOLS, CURLPROTO_HTTP | CURLPROTO_HTTPS);
2022-04-10 19:54:18 +03:00
if(is_array($opts) && count($opts) !== 0) {
2022-04-10 19:54:18 +03:00
Debug::log('Setting options: ' . json_encode($opts));
2022-04-10 19:54:18 +03:00
foreach($opts as $key => $value) {
curl_setopt($ch, $key, $value);
}
2022-04-10 19:54:18 +03:00
}
2022-04-10 19:54:18 +03:00
if(defined('PROXY_URL') && !defined('NOPROXY')) {
2022-04-10 19:54:18 +03:00
Debug::log('Setting proxy url: ' . PROXY_URL);
curl_setopt($ch, CURLOPT_PROXY, PROXY_URL);
2022-04-10 19:54:18 +03:00
}
2022-04-10 19:54:18 +03:00
// We always want the response header as part of the data!
curl_setopt($ch, CURLOPT_HEADER, true);
2022-04-10 19:54:18 +03:00
// Build "If-Modified-Since" header
if(!Debug::isEnabled() && $time = $cache->getTime()) { // Skip if cache file doesn't exist
Debug::log('Adding If-Modified-Since');
curl_setopt($ch, CURLOPT_TIMEVALUE, $time);
curl_setopt($ch, CURLOPT_TIMECONDITION, CURL_TIMECOND_IFMODSINCE);
}
2022-04-10 19:54:18 +03:00
// Enables logging for the outgoing header
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
2022-04-10 19:54:18 +03:00
$data = curl_exec($ch);
$errorCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
2022-04-10 19:54:18 +03:00
$curlError = curl_error($ch);
$curlErrno = curl_errno($ch);
$curlInfo = curl_getinfo($ch);
2022-04-10 19:54:18 +03:00
Debug::log('Outgoing header: ' . json_encode($curlInfo));
[contents] Replace file_get_contents by cURL cURL is a powerful library specifically designed to connect to many different types of servers with different types of protocols. For more detailed information refer to the PHP cURL manual: - http://php.net/manual/en/book.curl.php Due to this change some parameters for the getContents function were necessary (also applies to getSimpleHTMLDOM and getSimpleHTMLDOMCached): > $use_include_path removed This parameter has never been used and doesn't even make sense in this context; If set to true file_get_contents would also search for files in the include_path (specified in php.ini). > $context replaced by $header and $opts The $context parameter allowed for customization of the request in order to change how file_get_contents would acquire the data (i.e. using POST instead of GET, sending custom header, etc...) cURL also provides facilities to specify custom headers and change how it communicates to severs. cURL, however, is much more advanced. - $header is an optional parameter (empty by default). It receives an array of strings to send in the HTTP request header. See 'CURLOPT_HTTPHEADER': "An array of HTTP header fields to set, in the format array('Content-type: text/plain', 'Content-length: 100')" - php.net/manual/en/function.curl-setopt.php - $opts is an optional parameter (empty by default). It receives an array of options, where each option is a key-value-pair of a cURL option (CURLOPT_*) and it's associated parameter. This parameter accepts any of the CURLOPT_* settings. Example (sending POST instead of GET): $opts = array( CURLOPT_POST => 1, CURLOPT_POSTFIELDS => '&action=none' ); $html = getContents($url, array(), $opts); Refer to the cURL setopt manual for more information: - php.net/manual/en/function.curl-setopt.php > $offset and $maxlen removed These options were supported by file_get_contents, but there doesn't seem to be an equivalent in cURL. Since no caller uses them they are safe to remove. Compressed data / Encoding By using cURL instead of file_get_contents RSS-Bridge no longer has to handle compressed data manually. See 'CURLOPT_ENCODING': "[...] Supported encodings are "identity", "deflate", and "gzip". If an empty string, "", is set, a header containing all supported encoding types is sent." - http://php.net/manual/en/function.curl-setopt.php Notice: By default all encoding types are accepted (""). This can be changed by setting a custom option via $opts. Example: $opts = array(CURLOPT_ENCODING => 'gzip'); $html = getContents($url, array(), $opts); Proxy The proxy implementation should still work, but there doesn't seem to be an equivalent for 'request_fulluri = true'. To my understanding this isn't an issue because cURL knows how to handle proxy communication.
2018-03-25 14:12:13 +03:00
2022-04-10 19:54:18 +03:00
if($data === false)
Debug::log('Cant\'t download ' . $url . ' cUrl error: ' . $curlError . ' (' . $curlErrno . ')');
2022-04-10 19:54:18 +03:00
$headerSize = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$header = substr($data, 0, $headerSize);
$retVal['header'] = $header;
2022-04-10 19:54:18 +03:00
Debug::log('Response header: ' . $header);
2022-04-10 19:54:18 +03:00
$headers = parseResponseHeader($header);
$finalHeader = end($headers);
2022-04-10 19:54:18 +03:00
curl_close($ch);
$finalHeader = array_change_key_case($finalHeader, CASE_LOWER);
switch($errorCode) {
case 200: // Contents OK
case 201: // Contents Created
case 202: // Contents Accepted
Debug::log('New contents received');
$data = substr($data, $headerSize);
// Disable caching if the server responds with "Cache-Control: no-cache"
// or "Cache-Control: no-store"
if(array_key_exists('cache-control', $finalHeader)) {
Debug::log('Server responded with "Cache-Control" header');
$directives = explode(',', $finalHeader['cache-control']);
$directives = array_map('trim', $directives);
if(in_array('no-cache', $directives)
|| in_array('no-store', $directives)) { // Skip caching
Debug::log('Skip server side caching');
action: Add action to check bridge connectivity (#1147) * action: Add action to check bridge connectivity It is currently not simply possible to check if the remote server for a bridge is reachable or not, which means some of the bridges might no longer work because the server is no longer on the internet. In order to find those bridges we can either check each bridge individually (which takes a lot of effort), or use an automated script to do this for us. If a server is no longer reachable it could mean that it is temporarily unavailable, or shutdown permanently. The results of this script will at least help identifying such servers. * [Connectivity] Use Bootstrap container to properly display contents * [Connectivity] Limit connectivity checks to debug mode Connectivity checks take a long time to execute and can require a lot of bandwidth. Therefore, administrators should be able to determine when and who is able to utilize this action. The best way to prevent regular users from accessing this action is by making it available in debug mode only (public servers should never run in debug mode anyway). * [Connectivity] Split implemenation into multiple files * [Connectivity] Make web page responsive to user input * [Connectivity] Make status message sticky * [Connectivity] Add icon to the status message * [contents] Add the ability for getContents to return header information * [Connectivity] Add header information to the reply Json data * [Connectivity] Add new status (blue) for redirected sites Also adds titles to status icons (Successful, Redirected, Inactive, Failed) * [Connectivity] Fix show doesn't work for inactive bridges * [Connectivity] Fix typo * [Connectivity] Catch errors in promise chains * [Connectivity] Allow search by status and update dynamically * [Connectivity] Add a progress bar * [Connectivity] Use bridge factory * [Connectivity] Import Bootstrap v4.3.1 CSS
2019-11-01 00:02:38 +03:00
$retVal['content'] = $data;
break;
}
}
Debug::log('Store response to cache');
$cache->saveData($data);
action: Add action to check bridge connectivity (#1147) * action: Add action to check bridge connectivity It is currently not simply possible to check if the remote server for a bridge is reachable or not, which means some of the bridges might no longer work because the server is no longer on the internet. In order to find those bridges we can either check each bridge individually (which takes a lot of effort), or use an automated script to do this for us. If a server is no longer reachable it could mean that it is temporarily unavailable, or shutdown permanently. The results of this script will at least help identifying such servers. * [Connectivity] Use Bootstrap container to properly display contents * [Connectivity] Limit connectivity checks to debug mode Connectivity checks take a long time to execute and can require a lot of bandwidth. Therefore, administrators should be able to determine when and who is able to utilize this action. The best way to prevent regular users from accessing this action is by making it available in debug mode only (public servers should never run in debug mode anyway). * [Connectivity] Split implemenation into multiple files * [Connectivity] Make web page responsive to user input * [Connectivity] Make status message sticky * [Connectivity] Add icon to the status message * [contents] Add the ability for getContents to return header information * [Connectivity] Add header information to the reply Json data * [Connectivity] Add new status (blue) for redirected sites Also adds titles to status icons (Successful, Redirected, Inactive, Failed) * [Connectivity] Fix show doesn't work for inactive bridges * [Connectivity] Fix typo * [Connectivity] Catch errors in promise chains * [Connectivity] Allow search by status and update dynamically * [Connectivity] Add a progress bar * [Connectivity] Use bridge factory * [Connectivity] Import Bootstrap v4.3.1 CSS
2019-11-01 00:02:38 +03:00
$retVal['content'] = $data;
break;
case 304: // Not modified, use cached data
Debug::log('Contents not modified on host, returning cached data');
action: Add action to check bridge connectivity (#1147) * action: Add action to check bridge connectivity It is currently not simply possible to check if the remote server for a bridge is reachable or not, which means some of the bridges might no longer work because the server is no longer on the internet. In order to find those bridges we can either check each bridge individually (which takes a lot of effort), or use an automated script to do this for us. If a server is no longer reachable it could mean that it is temporarily unavailable, or shutdown permanently. The results of this script will at least help identifying such servers. * [Connectivity] Use Bootstrap container to properly display contents * [Connectivity] Limit connectivity checks to debug mode Connectivity checks take a long time to execute and can require a lot of bandwidth. Therefore, administrators should be able to determine when and who is able to utilize this action. The best way to prevent regular users from accessing this action is by making it available in debug mode only (public servers should never run in debug mode anyway). * [Connectivity] Split implemenation into multiple files * [Connectivity] Make web page responsive to user input * [Connectivity] Make status message sticky * [Connectivity] Add icon to the status message * [contents] Add the ability for getContents to return header information * [Connectivity] Add header information to the reply Json data * [Connectivity] Add new status (blue) for redirected sites Also adds titles to status icons (Successful, Redirected, Inactive, Failed) * [Connectivity] Fix show doesn't work for inactive bridges * [Connectivity] Fix typo * [Connectivity] Catch errors in promise chains * [Connectivity] Allow search by status and update dynamically * [Connectivity] Add a progress bar * [Connectivity] Use bridge factory * [Connectivity] Import Bootstrap v4.3.1 CSS
2019-11-01 00:02:38 +03:00
$retVal['content'] = $cache->loadData();
break;
default:
if(array_key_exists('server', $finalHeader) && stripos($finalHeader['server'], 'cloudflare') !== false) {
throw new CloudflareChallengeException($errorCode);
}
if ($curlError || $curlErrno) {
throw new GetContentsException('cURL error: ' . $curlError . ' (' . $curlErrno . ')');
}
throw new UnexpectedResponseException($retVal['content'], $retVal['header'], $errorCode);
}
action: Add action to check bridge connectivity (#1147) * action: Add action to check bridge connectivity It is currently not simply possible to check if the remote server for a bridge is reachable or not, which means some of the bridges might no longer work because the server is no longer on the internet. In order to find those bridges we can either check each bridge individually (which takes a lot of effort), or use an automated script to do this for us. If a server is no longer reachable it could mean that it is temporarily unavailable, or shutdown permanently. The results of this script will at least help identifying such servers. * [Connectivity] Use Bootstrap container to properly display contents * [Connectivity] Limit connectivity checks to debug mode Connectivity checks take a long time to execute and can require a lot of bandwidth. Therefore, administrators should be able to determine when and who is able to utilize this action. The best way to prevent regular users from accessing this action is by making it available in debug mode only (public servers should never run in debug mode anyway). * [Connectivity] Split implemenation into multiple files * [Connectivity] Make web page responsive to user input * [Connectivity] Make status message sticky * [Connectivity] Add icon to the status message * [contents] Add the ability for getContents to return header information * [Connectivity] Add header information to the reply Json data * [Connectivity] Add new status (blue) for redirected sites Also adds titles to status icons (Successful, Redirected, Inactive, Failed) * [Connectivity] Fix show doesn't work for inactive bridges * [Connectivity] Fix typo * [Connectivity] Catch errors in promise chains * [Connectivity] Allow search by status and update dynamically * [Connectivity] Add a progress bar * [Connectivity] Use bridge factory * [Connectivity] Import Bootstrap v4.3.1 CSS
2019-11-01 00:02:38 +03:00
return ($returnHeader === true) ? $retVal : $retVal['content'];
}
2018-11-16 23:48:59 +03:00
/**
* Gets contents from the Internet as simplhtmldom object.
*
* @param string $url The URL.
* @param array $header (optional) A list of cURL header.
* For more information follow the links below.
* * https://php.net/manual/en/function.curl-setopt.php
* * https://curl.haxx.se/libcurl/c/CURLOPT_HTTPHEADER.html
* @param array $opts (optional) A list of cURL options as associative array in
* the format `$opts[$option] = $value;`, where `$option` is any `CURLOPT_XXX`
* option and `$value` the corresponding value.
*
* For more information see http://php.net/manual/en/function.curl-setopt.php
* @param bool $lowercase Force all selectors to lowercase.
* @param bool $forceTagsClosed Forcefully close tags in malformed HTML.
*
* _Remarks_: Forcefully closing tags is great for malformed HTML, but it can
* lead to parsing errors.
* @param string $target_charset Defines the target charset.
* @param bool $stripRN Replace all occurrences of `"\r"` and `"\n"` by `" "`.
* @param string $defaultBRText Specifies the replacement text for `<br>` tags
* when returning plaintext.
* @param string $defaultSpanText Specifies the replacement text for `<span />`
* tags when returning plaintext.
* @return false|simple_html_dom Contents as simplehtmldom object.
2018-11-16 23:48:59 +03:00
*/
2017-02-14 19:28:07 +03:00
function getSimpleHTMLDOM($url,
$header = array(),
$opts = array(),
$lowercase = true,
$forceTagsClosed = true,
$target_charset = DEFAULT_TARGET_CHARSET,
$stripRN = true,
$defaultBRText = DEFAULT_BR_TEXT,
$defaultSpanText = DEFAULT_SPAN_TEXT){
[contents] Replace file_get_contents by cURL cURL is a powerful library specifically designed to connect to many different types of servers with different types of protocols. For more detailed information refer to the PHP cURL manual: - http://php.net/manual/en/book.curl.php Due to this change some parameters for the getContents function were necessary (also applies to getSimpleHTMLDOM and getSimpleHTMLDOMCached): > $use_include_path removed This parameter has never been used and doesn't even make sense in this context; If set to true file_get_contents would also search for files in the include_path (specified in php.ini). > $context replaced by $header and $opts The $context parameter allowed for customization of the request in order to change how file_get_contents would acquire the data (i.e. using POST instead of GET, sending custom header, etc...) cURL also provides facilities to specify custom headers and change how it communicates to severs. cURL, however, is much more advanced. - $header is an optional parameter (empty by default). It receives an array of strings to send in the HTTP request header. See 'CURLOPT_HTTPHEADER': "An array of HTTP header fields to set, in the format array('Content-type: text/plain', 'Content-length: 100')" - php.net/manual/en/function.curl-setopt.php - $opts is an optional parameter (empty by default). It receives an array of options, where each option is a key-value-pair of a cURL option (CURLOPT_*) and it's associated parameter. This parameter accepts any of the CURLOPT_* settings. Example (sending POST instead of GET): $opts = array( CURLOPT_POST => 1, CURLOPT_POSTFIELDS => '&action=none' ); $html = getContents($url, array(), $opts); Refer to the cURL setopt manual for more information: - php.net/manual/en/function.curl-setopt.php > $offset and $maxlen removed These options were supported by file_get_contents, but there doesn't seem to be an equivalent in cURL. Since no caller uses them they are safe to remove. Compressed data / Encoding By using cURL instead of file_get_contents RSS-Bridge no longer has to handle compressed data manually. See 'CURLOPT_ENCODING': "[...] Supported encodings are "identity", "deflate", and "gzip". If an empty string, "", is set, a header containing all supported encoding types is sent." - http://php.net/manual/en/function.curl-setopt.php Notice: By default all encoding types are accepted (""). This can be changed by setting a custom option via $opts. Example: $opts = array(CURLOPT_ENCODING => 'gzip'); $html = getContents($url, array(), $opts); Proxy The proxy implementation should still work, but there doesn't seem to be an equivalent for 'request_fulluri = true'. To my understanding this isn't an issue because cURL knows how to handle proxy communication.
2018-03-25 14:12:13 +03:00
$content = getContents($url, $header, $opts);
2017-02-14 19:28:07 +03:00
return str_get_html($content,
$lowercase,
$forceTagsClosed,
$target_charset,
$stripRN,
$defaultBRText,
$defaultSpanText);
}
/**
2018-11-16 23:48:59 +03:00
* Gets contents from the Internet as simplhtmldom object. Contents are cached
* and re-used for subsequent calls until the cache duration elapsed.
*
* _Notice_: Cached contents are forcefully removed after 24 hours (86400 seconds).
*
* @param string $url The URL.
* @param int $duration Cache duration in seconds.
* @param array $header (optional) A list of cURL header.
* For more information follow the links below.
* * https://php.net/manual/en/function.curl-setopt.php
* * https://curl.haxx.se/libcurl/c/CURLOPT_HTTPHEADER.html
* @param array $opts (optional) A list of cURL options as associative array in
* the format `$opts[$option] = $value;`, where `$option` is any `CURLOPT_XXX`
* option and `$value` the corresponding value.
*
* For more information see http://php.net/manual/en/function.curl-setopt.php
* @param bool $lowercase Force all selectors to lowercase.
* @param bool $forceTagsClosed Forcefully close tags in malformed HTML.
*
* _Remarks_: Forcefully closing tags is great for malformed HTML, but it can
* lead to parsing errors.
* @param string $target_charset Defines the target charset.
* @param bool $stripRN Replace all occurrences of `"\r"` and `"\n"` by `" "`.
* @param string $defaultBRText Specifies the replacement text for `<br>` tags
* when returning plaintext.
* @param string $defaultSpanText Specifies the replacement text for `<span />`
* tags when returning plaintext.
* @return false|simple_html_dom Contents as simplehtmldom object.
*/
2017-02-14 19:28:07 +03:00
function getSimpleHTMLDOMCached($url,
$duration = 86400,
$header = array(),
$opts = array(),
$lowercase = true,
$forceTagsClosed = true,
$target_charset = DEFAULT_TARGET_CHARSET,
$stripRN = true,
$defaultBRText = DEFAULT_BR_TEXT,
$defaultSpanText = DEFAULT_SPAN_TEXT){
Debug::log('Caching url ' . $url . ', duration ' . $duration);
// Initialize cache
$cacheFac = new CacheFactory();
$cacheFac->setWorkingDir(PATH_LIB_CACHES);
$cache = $cacheFac->create(Configuration::getConfig('cache', 'type'));
$cache->setScope('pages');
$cache->purgeCache(86400); // 24 hours (forced)
$params = array($url);
$cache->setKey($params);
// Determine if cached file is within duration
$time = $cache->getTime();
if($time !== false
&& (time() - $duration < $time)
&& !Debug::isEnabled()) { // Contents within duration
$content = $cache->loadData();
} else { // Content not within duration
[contents] Replace file_get_contents by cURL cURL is a powerful library specifically designed to connect to many different types of servers with different types of protocols. For more detailed information refer to the PHP cURL manual: - http://php.net/manual/en/book.curl.php Due to this change some parameters for the getContents function were necessary (also applies to getSimpleHTMLDOM and getSimpleHTMLDOMCached): > $use_include_path removed This parameter has never been used and doesn't even make sense in this context; If set to true file_get_contents would also search for files in the include_path (specified in php.ini). > $context replaced by $header and $opts The $context parameter allowed for customization of the request in order to change how file_get_contents would acquire the data (i.e. using POST instead of GET, sending custom header, etc...) cURL also provides facilities to specify custom headers and change how it communicates to severs. cURL, however, is much more advanced. - $header is an optional parameter (empty by default). It receives an array of strings to send in the HTTP request header. See 'CURLOPT_HTTPHEADER': "An array of HTTP header fields to set, in the format array('Content-type: text/plain', 'Content-length: 100')" - php.net/manual/en/function.curl-setopt.php - $opts is an optional parameter (empty by default). It receives an array of options, where each option is a key-value-pair of a cURL option (CURLOPT_*) and it's associated parameter. This parameter accepts any of the CURLOPT_* settings. Example (sending POST instead of GET): $opts = array( CURLOPT_POST => 1, CURLOPT_POSTFIELDS => '&action=none' ); $html = getContents($url, array(), $opts); Refer to the cURL setopt manual for more information: - php.net/manual/en/function.curl-setopt.php > $offset and $maxlen removed These options were supported by file_get_contents, but there doesn't seem to be an equivalent in cURL. Since no caller uses them they are safe to remove. Compressed data / Encoding By using cURL instead of file_get_contents RSS-Bridge no longer has to handle compressed data manually. See 'CURLOPT_ENCODING': "[...] Supported encodings are "identity", "deflate", and "gzip". If an empty string, "", is set, a header containing all supported encoding types is sent." - http://php.net/manual/en/function.curl-setopt.php Notice: By default all encoding types are accepted (""). This can be changed by setting a custom option via $opts. Example: $opts = array(CURLOPT_ENCODING => 'gzip'); $html = getContents($url, array(), $opts); Proxy The proxy implementation should still work, but there doesn't seem to be an equivalent for 'request_fulluri = true'. To my understanding this isn't an issue because cURL knows how to handle proxy communication.
2018-03-25 14:12:13 +03:00
$content = getContents($url, $header, $opts);
if($content !== false) {
$cache->saveData($content);
}
}
2017-02-14 19:28:07 +03:00
return str_get_html($content,
$lowercase,
$forceTagsClosed,
$target_charset,
$stripRN,
$defaultBRText,
$defaultSpanText);
}
/**
2018-11-16 23:48:59 +03:00
* Parses the cURL response header into an associative array
*
* Based on https://stackoverflow.com/a/18682872
2018-11-16 23:48:59 +03:00
*
* @param string $header The cURL response header.
* @return array An associative array of response headers.
*/
function parseResponseHeader($header) {
$headers = array();
$requests = explode("\r\n\r\n", trim($header));
foreach ($requests as $request) {
$header = array();
foreach (explode("\r\n", $request) as $i => $line) {
if($i === 0) {
$header['http_code'] = $line;
} else {
list ($key, $value) = explode(':', $line);
$header[$key] = trim($value);
}
}
$headers[] = $header;
}
return $headers;
}
Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802) * Debug mode improvements - Improve debug warning message - Restore error reporting in debug mode - Fix 'notice' messages for unset fields * Add parsing utility functions html.php - extractFromDelimiters - stripWithDelimiters - stripRecursiveHTMLSection - markdownToHtml (partial) bridges - remove now-duplicate functions - call functions from html.php instead * [Anidex] New bridge Anime torrent tracker * [Anime-Ultime] Restore thumbnail * [CNET] Recreate bridge Full rewrite as the previous one was broken * [Dilbert] Minor URI fix Use new self::URI property * [EstCeQuonMetEnProd] Fix content extraction Bridge was broken * [Facebook] Fix "SpSonsSoriSsés" label ... which was taking space in item title * [Futura-Sciences] Use HTTPS, More cleanup Use HTTPS as FS now offer HTTPS Clean additional useless HTML elements * [GBATemp] Multiple fixes - Fix categories: missing "break" statements - Restore thumbnail as enclosure - Fix date extraction - Fix user blog post extraction - Use getSimpleHTMLDOMCached * [JapanExpo] Fix bridge, HTTPS, thumbnails - Fix getSimpleHTMLDOMCached call - Upgrade to HTTPS as JE now offers HTTPS - Restore thumbnails as enclosures * [LeMondeInformatique] Fix bridge, HTTPS - Upgrade to HTTPS as LMI now offers HTTPS - Restore thumbnails using small images - Fix content extraction - Fix text encoding issue * [Nextgov] Fix content extraction - Restore thumbnail and use small image - Field extraction fixes * [NextInpact] Add categories and filtering by type - Offer all RSS feeds - Allow filtering by article type - Implement extraction for brief articles - Remove article limit, many brief articles are publied all at once * [NyaaTorrents] New bridge Anime torrent tracker * [Releases3DS] Cache content, restore thumbnail - Use getSimpleHTMLDOMCached - Restore thumbnail as enclosure * [TheHackerNews] Fix bridge - Fix content extraction including article body - Restore thumbnail as enclosure * [WeLiveSecurity] HTTPS, Fix content extraction - Upgrade to HTTPS as WLS now offers HTTPS - Fix content extraction including article body * [WordPress] Reduce timeout, more content selectors - Reduce timeout to use default one (1h) - Add new content selector (articleBody) - Find thumbnail and set as enclosure - Fix <script> cleanup * [YGGTorrent] Increase limit, use cache - Increase item limit as uploads are very frequent - Use getSimpleHTMLDOMCached * [ZDNet] Rewrite with FeedExpander - Upgrade to HTTPS as ZD now offers HTTPS - Use FeedExpander for secondary fields - Fix content extraction for article body * [Main] Handle MIME type for enclosures Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content). One can force enclosure type using #.ext anchor (hacky, needs improving) * [FeedExpander] Improve field extraction - Add support for passing enclosures - Improve author and uri extraction - Fix 'notice' PHP error messages * [Pull] Coding style fixes for #802 * [Pull] Implementing changes for #802 - Fix coding style issues with str append - Remove useless CACHE_TIMEOUT - Use count() instead of $limit - Use defaultLinkTo() + handle strings - Use http_build_query() - Fix missing </em> - Remove error_reporting(0) - warning CSS (@LogMANOriginal) - Fix typo in FeedExpander comment * [Main] More documentation for markdownToHtml See #802 for more details
2018-09-09 22:20:13 +03:00
/**
2018-11-16 23:48:59 +03:00
* Determines the MIME type from a URL/Path file extension.
*
* _Remarks_:
*
* * The built-in functions `mime_content_type` and `fileinfo` require fetching
* remote contents.
* * A caller can hint for a MIME type by appending `#.ext` to the URL (i.e. `#.image`).
*
Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802) * Debug mode improvements - Improve debug warning message - Restore error reporting in debug mode - Fix 'notice' messages for unset fields * Add parsing utility functions html.php - extractFromDelimiters - stripWithDelimiters - stripRecursiveHTMLSection - markdownToHtml (partial) bridges - remove now-duplicate functions - call functions from html.php instead * [Anidex] New bridge Anime torrent tracker * [Anime-Ultime] Restore thumbnail * [CNET] Recreate bridge Full rewrite as the previous one was broken * [Dilbert] Minor URI fix Use new self::URI property * [EstCeQuonMetEnProd] Fix content extraction Bridge was broken * [Facebook] Fix "SpSonsSoriSsés" label ... which was taking space in item title * [Futura-Sciences] Use HTTPS, More cleanup Use HTTPS as FS now offer HTTPS Clean additional useless HTML elements * [GBATemp] Multiple fixes - Fix categories: missing "break" statements - Restore thumbnail as enclosure - Fix date extraction - Fix user blog post extraction - Use getSimpleHTMLDOMCached * [JapanExpo] Fix bridge, HTTPS, thumbnails - Fix getSimpleHTMLDOMCached call - Upgrade to HTTPS as JE now offers HTTPS - Restore thumbnails as enclosures * [LeMondeInformatique] Fix bridge, HTTPS - Upgrade to HTTPS as LMI now offers HTTPS - Restore thumbnails using small images - Fix content extraction - Fix text encoding issue * [Nextgov] Fix content extraction - Restore thumbnail and use small image - Field extraction fixes * [NextInpact] Add categories and filtering by type - Offer all RSS feeds - Allow filtering by article type - Implement extraction for brief articles - Remove article limit, many brief articles are publied all at once * [NyaaTorrents] New bridge Anime torrent tracker * [Releases3DS] Cache content, restore thumbnail - Use getSimpleHTMLDOMCached - Restore thumbnail as enclosure * [TheHackerNews] Fix bridge - Fix content extraction including article body - Restore thumbnail as enclosure * [WeLiveSecurity] HTTPS, Fix content extraction - Upgrade to HTTPS as WLS now offers HTTPS - Fix content extraction including article body * [WordPress] Reduce timeout, more content selectors - Reduce timeout to use default one (1h) - Add new content selector (articleBody) - Find thumbnail and set as enclosure - Fix <script> cleanup * [YGGTorrent] Increase limit, use cache - Increase item limit as uploads are very frequent - Use getSimpleHTMLDOMCached * [ZDNet] Rewrite with FeedExpander - Upgrade to HTTPS as ZD now offers HTTPS - Use FeedExpander for secondary fields - Fix content extraction for article body * [Main] Handle MIME type for enclosures Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content). One can force enclosure type using #.ext anchor (hacky, needs improving) * [FeedExpander] Improve field extraction - Add support for passing enclosures - Improve author and uri extraction - Fix 'notice' PHP error messages * [Pull] Coding style fixes for #802 * [Pull] Implementing changes for #802 - Fix coding style issues with str append - Remove useless CACHE_TIMEOUT - Use count() instead of $limit - Use defaultLinkTo() + handle strings - Use http_build_query() - Fix missing </em> - Remove error_reporting(0) - warning CSS (@LogMANOriginal) - Fix typo in FeedExpander comment * [Main] More documentation for markdownToHtml See #802 for more details
2018-09-09 22:20:13 +03:00
* Based on https://stackoverflow.com/a/1147952
2018-11-16 23:48:59 +03:00
*
* @param string $url The URL or path to the file.
* @return string The MIME type of the file.
Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802) * Debug mode improvements - Improve debug warning message - Restore error reporting in debug mode - Fix 'notice' messages for unset fields * Add parsing utility functions html.php - extractFromDelimiters - stripWithDelimiters - stripRecursiveHTMLSection - markdownToHtml (partial) bridges - remove now-duplicate functions - call functions from html.php instead * [Anidex] New bridge Anime torrent tracker * [Anime-Ultime] Restore thumbnail * [CNET] Recreate bridge Full rewrite as the previous one was broken * [Dilbert] Minor URI fix Use new self::URI property * [EstCeQuonMetEnProd] Fix content extraction Bridge was broken * [Facebook] Fix "SpSonsSoriSsés" label ... which was taking space in item title * [Futura-Sciences] Use HTTPS, More cleanup Use HTTPS as FS now offer HTTPS Clean additional useless HTML elements * [GBATemp] Multiple fixes - Fix categories: missing "break" statements - Restore thumbnail as enclosure - Fix date extraction - Fix user blog post extraction - Use getSimpleHTMLDOMCached * [JapanExpo] Fix bridge, HTTPS, thumbnails - Fix getSimpleHTMLDOMCached call - Upgrade to HTTPS as JE now offers HTTPS - Restore thumbnails as enclosures * [LeMondeInformatique] Fix bridge, HTTPS - Upgrade to HTTPS as LMI now offers HTTPS - Restore thumbnails using small images - Fix content extraction - Fix text encoding issue * [Nextgov] Fix content extraction - Restore thumbnail and use small image - Field extraction fixes * [NextInpact] Add categories and filtering by type - Offer all RSS feeds - Allow filtering by article type - Implement extraction for brief articles - Remove article limit, many brief articles are publied all at once * [NyaaTorrents] New bridge Anime torrent tracker * [Releases3DS] Cache content, restore thumbnail - Use getSimpleHTMLDOMCached - Restore thumbnail as enclosure * [TheHackerNews] Fix bridge - Fix content extraction including article body - Restore thumbnail as enclosure * [WeLiveSecurity] HTTPS, Fix content extraction - Upgrade to HTTPS as WLS now offers HTTPS - Fix content extraction including article body * [WordPress] Reduce timeout, more content selectors - Reduce timeout to use default one (1h) - Add new content selector (articleBody) - Find thumbnail and set as enclosure - Fix <script> cleanup * [YGGTorrent] Increase limit, use cache - Increase item limit as uploads are very frequent - Use getSimpleHTMLDOMCached * [ZDNet] Rewrite with FeedExpander - Upgrade to HTTPS as ZD now offers HTTPS - Use FeedExpander for secondary fields - Fix content extraction for article body * [Main] Handle MIME type for enclosures Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content). One can force enclosure type using #.ext anchor (hacky, needs improving) * [FeedExpander] Improve field extraction - Add support for passing enclosures - Improve author and uri extraction - Fix 'notice' PHP error messages * [Pull] Coding style fixes for #802 * [Pull] Implementing changes for #802 - Fix coding style issues with str append - Remove useless CACHE_TIMEOUT - Use count() instead of $limit - Use defaultLinkTo() + handle strings - Use http_build_query() - Fix missing </em> - Remove error_reporting(0) - warning CSS (@LogMANOriginal) - Fix typo in FeedExpander comment * [Main] More documentation for markdownToHtml See #802 for more details
2018-09-09 22:20:13 +03:00
*/
function getMimeType($url) {
static $mime = null;
if (is_null($mime)) {
// Default values, overriden by /etc/mime.types when present
Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802) * Debug mode improvements - Improve debug warning message - Restore error reporting in debug mode - Fix 'notice' messages for unset fields * Add parsing utility functions html.php - extractFromDelimiters - stripWithDelimiters - stripRecursiveHTMLSection - markdownToHtml (partial) bridges - remove now-duplicate functions - call functions from html.php instead * [Anidex] New bridge Anime torrent tracker * [Anime-Ultime] Restore thumbnail * [CNET] Recreate bridge Full rewrite as the previous one was broken * [Dilbert] Minor URI fix Use new self::URI property * [EstCeQuonMetEnProd] Fix content extraction Bridge was broken * [Facebook] Fix "SpSonsSoriSsés" label ... which was taking space in item title * [Futura-Sciences] Use HTTPS, More cleanup Use HTTPS as FS now offer HTTPS Clean additional useless HTML elements * [GBATemp] Multiple fixes - Fix categories: missing "break" statements - Restore thumbnail as enclosure - Fix date extraction - Fix user blog post extraction - Use getSimpleHTMLDOMCached * [JapanExpo] Fix bridge, HTTPS, thumbnails - Fix getSimpleHTMLDOMCached call - Upgrade to HTTPS as JE now offers HTTPS - Restore thumbnails as enclosures * [LeMondeInformatique] Fix bridge, HTTPS - Upgrade to HTTPS as LMI now offers HTTPS - Restore thumbnails using small images - Fix content extraction - Fix text encoding issue * [Nextgov] Fix content extraction - Restore thumbnail and use small image - Field extraction fixes * [NextInpact] Add categories and filtering by type - Offer all RSS feeds - Allow filtering by article type - Implement extraction for brief articles - Remove article limit, many brief articles are publied all at once * [NyaaTorrents] New bridge Anime torrent tracker * [Releases3DS] Cache content, restore thumbnail - Use getSimpleHTMLDOMCached - Restore thumbnail as enclosure * [TheHackerNews] Fix bridge - Fix content extraction including article body - Restore thumbnail as enclosure * [WeLiveSecurity] HTTPS, Fix content extraction - Upgrade to HTTPS as WLS now offers HTTPS - Fix content extraction including article body * [WordPress] Reduce timeout, more content selectors - Reduce timeout to use default one (1h) - Add new content selector (articleBody) - Find thumbnail and set as enclosure - Fix <script> cleanup * [YGGTorrent] Increase limit, use cache - Increase item limit as uploads are very frequent - Use getSimpleHTMLDOMCached * [ZDNet] Rewrite with FeedExpander - Upgrade to HTTPS as ZD now offers HTTPS - Use FeedExpander for secondary fields - Fix content extraction for article body * [Main] Handle MIME type for enclosures Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content). One can force enclosure type using #.ext anchor (hacky, needs improving) * [FeedExpander] Improve field extraction - Add support for passing enclosures - Improve author and uri extraction - Fix 'notice' PHP error messages * [Pull] Coding style fixes for #802 * [Pull] Implementing changes for #802 - Fix coding style issues with str append - Remove useless CACHE_TIMEOUT - Use count() instead of $limit - Use defaultLinkTo() + handle strings - Use http_build_query() - Fix missing </em> - Remove error_reporting(0) - warning CSS (@LogMANOriginal) - Fix typo in FeedExpander comment * [Main] More documentation for markdownToHtml See #802 for more details
2018-09-09 22:20:13 +03:00
$mime = array(
'jpg' => 'image/jpeg',
'gif' => 'image/gif',
'png' => 'image/png',
'image' => 'image/*'
);
// '@' is used to mute open_basedir warning, see issue #818
if (@is_readable('/etc/mime.types')) {
Catching up | [Main] Debug mode, parse utils, MIME | [Bridges] Add/Improve 20 bridges (#802) * Debug mode improvements - Improve debug warning message - Restore error reporting in debug mode - Fix 'notice' messages for unset fields * Add parsing utility functions html.php - extractFromDelimiters - stripWithDelimiters - stripRecursiveHTMLSection - markdownToHtml (partial) bridges - remove now-duplicate functions - call functions from html.php instead * [Anidex] New bridge Anime torrent tracker * [Anime-Ultime] Restore thumbnail * [CNET] Recreate bridge Full rewrite as the previous one was broken * [Dilbert] Minor URI fix Use new self::URI property * [EstCeQuonMetEnProd] Fix content extraction Bridge was broken * [Facebook] Fix "SpSonsSoriSsés" label ... which was taking space in item title * [Futura-Sciences] Use HTTPS, More cleanup Use HTTPS as FS now offer HTTPS Clean additional useless HTML elements * [GBATemp] Multiple fixes - Fix categories: missing "break" statements - Restore thumbnail as enclosure - Fix date extraction - Fix user blog post extraction - Use getSimpleHTMLDOMCached * [JapanExpo] Fix bridge, HTTPS, thumbnails - Fix getSimpleHTMLDOMCached call - Upgrade to HTTPS as JE now offers HTTPS - Restore thumbnails as enclosures * [LeMondeInformatique] Fix bridge, HTTPS - Upgrade to HTTPS as LMI now offers HTTPS - Restore thumbnails using small images - Fix content extraction - Fix text encoding issue * [Nextgov] Fix content extraction - Restore thumbnail and use small image - Field extraction fixes * [NextInpact] Add categories and filtering by type - Offer all RSS feeds - Allow filtering by article type - Implement extraction for brief articles - Remove article limit, many brief articles are publied all at once * [NyaaTorrents] New bridge Anime torrent tracker * [Releases3DS] Cache content, restore thumbnail - Use getSimpleHTMLDOMCached - Restore thumbnail as enclosure * [TheHackerNews] Fix bridge - Fix content extraction including article body - Restore thumbnail as enclosure * [WeLiveSecurity] HTTPS, Fix content extraction - Upgrade to HTTPS as WLS now offers HTTPS - Fix content extraction including article body * [WordPress] Reduce timeout, more content selectors - Reduce timeout to use default one (1h) - Add new content selector (articleBody) - Find thumbnail and set as enclosure - Fix <script> cleanup * [YGGTorrent] Increase limit, use cache - Increase item limit as uploads are very frequent - Use getSimpleHTMLDOMCached * [ZDNet] Rewrite with FeedExpander - Upgrade to HTTPS as ZD now offers HTTPS - Use FeedExpander for secondary fields - Fix content extraction for article body * [Main] Handle MIME type for enclosures Many feed readers will ignore enclosures (e.g. thumbnails) with no MIME type. This commit adds automatic MIME type detection based on file extension (which may be inaccurate but is the only way without fetching the content). One can force enclosure type using #.ext anchor (hacky, needs improving) * [FeedExpander] Improve field extraction - Add support for passing enclosures - Improve author and uri extraction - Fix 'notice' PHP error messages * [Pull] Coding style fixes for #802 * [Pull] Implementing changes for #802 - Fix coding style issues with str append - Remove useless CACHE_TIMEOUT - Use count() instead of $limit - Use defaultLinkTo() + handle strings - Use http_build_query() - Fix missing </em> - Remove error_reporting(0) - warning CSS (@LogMANOriginal) - Fix typo in FeedExpander comment * [Main] More documentation for markdownToHtml See #802 for more details
2018-09-09 22:20:13 +03:00
$file = fopen('/etc/mime.types', 'r');
while(($line = fgets($file)) !== false) {
$line = trim(preg_replace('/#.*/', '', $line));
if(!$line)
continue;
$parts = preg_split('/\s+/', $line);
if(count($parts) == 1)
continue;
$type = array_shift($parts);
foreach($parts as $part)
$mime[$part] = $type;
}
fclose($file);
}
}
if (strpos($url, '?') !== false) {
$url_temp = substr($url, 0, strpos($url, '?'));
if (strpos($url, '#') !== false) {
$anchor = substr($url, strpos($url, '#'));
$url_temp .= $anchor;
}
$url = $url_temp;
}
$ext = strtolower(pathinfo($url, PATHINFO_EXTENSION));
if (!empty($mime[$ext])) {
return $mime[$ext];
}
return 'application/octet-stream';
}