Commit graph

659 commits

Author SHA1 Message Date
Dag
daef240cd2
test: add test for FeedParser (#3754) 2023-10-13 23:14:08 +02:00
Dag
5f37c72be0
fix(binance): plus some other tweaks (#3753) 2023-10-13 20:48:08 +02:00
Dag
920d00480d
fix(senscritique) (#3750) 2023-10-13 11:24:22 +02:00
Dag
49d9dafaec
refactor: more feed parsing tweaks (#3748) 2023-10-13 02:31:09 +02:00
Dag
2880524dfc
refactor: remove parent calls to parseItem (#3747) 2023-10-13 01:59:05 +02:00
Dag
e379019db2
refactor (#3746) 2023-10-13 01:02:19 +02:00
Dag
44fb2c98bc
fix: various fixes (#3745) 2023-10-13 00:26:11 +02:00
Dag
9bda9e246a
refactor: FeedExpander (#3740)
* refactor: FeedExpander
2023-10-12 22:14:04 +02:00
Dag
6a72c56cdd
fix: various fixes (#3741) 2023-10-12 19:49:04 +02:00
Dag
b6a9baff94
fix(cvedetails,tldrtech) (#3735) 2023-10-10 21:41:57 +02:00
User123698745
59dd49671d
[BridgeCard] add example value to info hint and allow using it by right click (#3726) 2023-10-02 03:02:57 +02:00
Dag
547af0d0d2
refactor: use Json::encode instead of json_encode (#3724) 2023-10-01 20:54:28 +02:00
User123698745
69da0dd583
[refactoring] replace direct use of curl with getContents (#3723)
+ some fixed warnings
2023-10-01 20:46:51 +02:00
Dag
41df17bc46
refactor (#3712)
* test: refactor test suite

* docs

* refactor

* yup

* docs
2023-10-01 19:23:30 +02:00
Dag
7273a05f02
fix: google play and tiktok (#3722)
* fix(googleplay)

* fix(tiktok)
2023-10-01 18:53:50 +02:00
Dag
ae53adefad
refactor: FeedItem::setTimestamp() (#3711) 2023-09-26 00:27:45 +02:00
Dag
f421c45b21
test: add feed item test (#3709)
* test: add feed item test

also some refactor

* yup

* yup
2023-09-25 22:32:15 +02:00
Dag
cd30c25b08
refactor (#3708) 2023-09-25 21:18:48 +02:00
ORelio
e1b911fc1f
[CssSelectorBridge] Retrieve metadata for social media embeds (#3602, #3687) (#3706)
* [CssSelectorBridge] Metadata from social embed (#3602, #3687)

Implement the following metadata sources:
 - Facebook Open Graph
 - Twitter <meta> tags
 - Standard <meta> tags
 - JSON linked data (ld+json)

The following metadata is supported:
 - Canonical URL (may help removing garbage from URLs)
 - Article title
 - Truncated summary
 - Published/Updated timestamp
 - Enclosure/Thumbnail image
 - Author Name or Twitter handle

SitemapBridge will also automatically benefit from this commit.

* [php8backports] Add array_is_list()

Needed this function for ld+json implementation in CssSelectorBridge.

* [SitemapBridge] Add option to discard thumbnail

* [CssSelectorBridge] Fix linting issues
2023-09-24 23:07:43 +02:00
Dag
857e908929
chore: prepare 2023-09-24 release (#3703) 2023-09-24 20:53:07 +02:00
Dag
f321f000c1
feat: add url component (#3684)
* feat: add url library

* fix
2023-09-24 18:34:09 +02:00
Dag
0c69148cff
fix(vice): news rss changed (#3694)
* fix: typo in prior commit

* fix(vice): news rss changed
2023-09-23 20:39:02 +02:00
Dag
b3b0736761
feat: improve error/exception ui (#3690) 2023-09-23 18:54:14 +02:00
Dag
07f49225d9
fix: bug in refactor (#3688) 2023-09-23 16:52:39 +02:00
Dag
7329b83cc0
refactor: logger (#3678) 2023-09-21 22:05:55 +02:00
Dag
e6aef73a02
refactor (#3668) 2023-09-20 02:45:48 +02:00
User123698745
3b91b1d260
[XPathBridge] add option to skip htmlspecialchars (#3672) 2023-09-15 01:58:06 +02:00
mruac
4f5a492dde
[BridgeAbstract] fix undefined index issue (#3665)
* .

* attempt to fix #2943
https://github.com/RSS-Bridge/rss-bridge/issues/2943

* Revert "."

This reverts commit c0b6ccfea6.

* lint

* Revert "attempt to fix #2943"

This reverts commit 9f1a66e48d.

* moved fix to BridgeAbstract

* fix undefined index

* lint
2023-09-11 13:18:00 +02:00
Dag
4b9f6f7e53
fix: rewrite and improve caching (#3594) 2023-09-10 21:50:15 +02:00
mruac
a786bbd4e0
DisplayAction: defaultchecked fix (#3654)
* .

* attempt to fix #2943
https://github.com/RSS-Bridge/rss-bridge/issues/2943

* Revert "."

This reverts commit c0b6ccfea6.

* lint

* Revert "attempt to fix #2943"

This reverts commit 9f1a66e48d.

* moved fix to BridgeAbstract
2023-09-10 04:15:05 +02:00
csisoap
dbe37cc302
[TwitterBridge] Filter out any promoted tweet (#3652)
* Filter out any advertise tweet

* Make some filter work, fix bug that may happen with tweet id list.

* clear phpcs warning, ignore line length warning
2023-09-06 16:14:11 +02:00
csisoap
4d05d0beff
[TwitterBridge] Add support for OAuth authorization. (#3628)
* Update TwitterClient.php

- Add OAuth authorization header.
- Add new endpoint.

* Update TwitterBridge.php

- Make some changes to support new endpoint.

* Update TwitterBridge.php

* clean up, fix warning

* fix warning

* fix warning

* remove oauth token

* fix wrong twitter id when encounter reply post.

* Update TwitterClient.php

* fix wrong twitter id cause by previous commit

* clear warning

* attempt to clear warning

* attempt to clear warning
2023-08-29 17:14:34 +02:00
User123698745
4976cd227e
[FeedExpander] support xhtml content / content with child elements (#3598)
* [core] support xhtml content type in FeedExpander

* [FilterBridge] change defaultValue to exampleValue

* [core] support content with child elements in FeedExpander
2023-08-04 22:14:08 +02:00
User123698745
7661a78a43
[core] add bridge not found warning message to frontpage (#3591) 2023-08-03 03:10:24 +02:00
Dag
ed97ce8646
fix: dont fail for non-existing enabled bridge (#3589)
* fix: dont fail for non-existing enabled bridge

* yup
2023-08-01 19:35:15 +02:00
mruac
10f7b6f4f6
Fix php8.2 deprecated warning when using bridge specific configurations (#3587)
* Fix php8.2 deprecated warning

Fix php8.2 warning: `Deprecated: Creation of dynamic property is deprecated`

* fix

* refactor: remove unused method

---------

Co-authored-by: Dag <me@dvikan.no>
2023-08-01 19:35:06 +02:00
Dag
8e2353ad3e
fix: write to cache only if data is was not cached, fix #3586 (#3588) 2023-08-01 06:19:42 +02:00
Dag
7e4807530e
fix: various small fixes (#3580) 2023-07-31 20:43:18 +02:00
User123698745
93eecdf79f
[core] fix new bridge PRs not generating html preview artifacts (#3583)
* [core] replace everything except bridge name to get a valid whitelist.txt

* [core] do not use hard code repository name to improve working with forks

* [core] trim bridge names from whitelist.txt to reduce chance of failure
2023-07-30 23:26:59 +02:00
Dag
701fe3cfed
fix: various small fixes (#3578) 2023-07-29 00:14:30 +02:00
csisoap
556bca58cf
[TwitterBridge] Fix search, user, list ID (#3566)
* Add ability to fetch user, list tweet

* Fix user, search, list ID although list still broke

* clear whitespace

* Revert CACHE_TIMEOUT

* clear whitespace, change single quote

* Clear PHP warning, add ability to get full-text if truncated

* Clear PHP warning

* clear warning

* clear whitespace

* Add check condition for mediaDetails.

* Add whitespace

* Add try catch exception for get full-text tweet

* clear warning

* clear warning
2023-07-25 22:36:41 +02:00
Dag
74635fd752
fix(DisplayAction): improve error handling and cache logic (#3558)
* fix(DisplayAction): improve error handling and cache logic

* restore prev timeouts

* refactor

* yup

* test: fix unit test

* leave twitter client unchanged

* leave twitter bridge unchanged
2023-07-23 23:05:35 +02:00
Dag
517c7f5c9b
fix(cache): bug (#3554) 2023-07-19 22:18:42 +02:00
Dag
93620aa105
fix(cache): bug in cache logic (#3553)
It is possible to have a cached item with a very old mtime but it's technically expired.

So, check for presence of time and whether the time it is within 10 days
2023-07-19 22:05:26 +02:00
Dag
6254b8593e
refactor(cache): extract and encapsulate cache expiration logic (#3547)
* refactor(cache): extract and encapsulate cache expiration logic

* fix: logic bug in getSimpleHTMLDOMCached

* fix: silly me, index should of course be on the key column

* silly me again, PRIMARY keys get index by default lol

* comment out the delete portion in loadData

* remove a few log statements

* tweak twitter cache timeout
2023-07-19 05:05:49 +02:00
Dag
08d16322e1
fix: bug in prior conflict merge (#3546) 2023-07-16 22:37:37 +02:00
Dag
a59793e8d6
refactor: extract CurlHttpClient (#3532)
* refactor: extract CurlHttpClient

* refactor

* interface
2023-07-16 22:07:34 +02:00
Dag
310160fd92
feat: improve http 429 handling (#3541) 2023-07-16 07:18:38 +02:00
Dag
eaea8e6640
fix: Undefined index: REMOTE_ADDR in non-web sapi' (#3538) 2023-07-15 21:16:23 +02:00
Fake4d
0f2b55fbef
Update Configuration.php - Old Version Date in new Release (#3526)
Old Version Date in new Release
2023-07-12 11:56:07 +02:00
Dag
69aa751f40
fix(cache): bug in prior refactor (#3525)
* fix(cache): bug in prior refactor

* yup
2023-07-11 23:26:22 +02:00
csisoap
b3bf95bfdd
Replace token for Twitter. (#3522) 2023-07-11 23:17:38 +02:00
Dag
6c0e186d3f
fix(jornaln): Array to string conversion at lib/BridgeAbstract.php li… (#3523)
* fix(jornaln): Array to string conversion at lib/BridgeAbstract.php line 320

* yup
2023-07-11 16:54:59 +02:00
Dag
c9a861e259
fix(cache): bug in prior refactor (#3520) 2023-07-09 15:24:29 +02:00
Dag
dfe78fb379
fix: various small fixes (#3519) 2023-07-09 10:08:30 +02:00
Dag
7881c87bed
fix: various small fixes (#3517)
* fix(asrocknews): Trying to get property src of non-object

Trying to get property 'src' of non-object at bridges/ASRockNewsBridge.php line 37

* refactor(http): tweak max redirs config

* fix(tiktok)

* fix(gizmodo)

* fix(craig)

* fix(nationalg)

* fix(roadandtrack)

* fix(etsy)
2023-07-08 23:21:55 +02:00
Dag
91976f7d56
fix(file cache): acquire lock before writing (#3509) 2023-07-08 17:06:49 +02:00
Dag
c1c8304fc0
refactor: dont create multiple instances of the cache (#3504) 2023-07-08 17:03:12 +02:00
Dag
b594ad2de3
fix: discard empty lines in whitelist.txt (#3507) 2023-07-07 11:25:36 +02:00
Dag
5e22459eb6
fix: remove unnecessary calls to purgeCache (#3502) 2023-07-06 18:52:19 +02:00
Dag
965d7d44c5
feat(sqlite cache): add config options (#3499)
* refactor: sqlite cache

* refactor

* feat: add config options to sqlite cache

* refactor
2023-07-06 15:59:38 +02:00
Dag
caac7f572c
refacor: improve cache interface (#3492)
* fix: proper typehint on setScope

* refactor: type hint setKey()

* typehint
2023-07-06 15:10:30 +02:00
Dag
a9fd3b9e61
fix(CacheInterface): logic bug in getTime (#3491)
* fix(CacheInterface): logic bug in getTime

* test
2023-07-05 17:37:21 +02:00
Dag
a21d496bc7
feat: add default arg to Configuration::getConfig (#3331) 2023-07-05 05:33:22 +02:00
Dag
bf73372d7f
fix: dont be case-sensitive on env vars (#3475) 2023-07-02 06:47:21 +02:00
Dag
748fc9fd65
fix: various small notice fixes (#3474)
* fix(patreon): php notice

* fix(pepperbridge): php notice

* fix(ebay): php notice

* fix(tiktok): php notice

* fix(yandex): fix notice

* fix(justwatch): notice

* lint
2023-07-02 06:40:25 +02:00
Dag
372880b5ef
fix: file cache tweaks (#3470)
* fix: improve file cache

* fix(filecache): log when unserialize fails
2023-06-30 22:31:19 +02:00
Dag
8eabdbe5f8
fix(Twitter): properly find time line entries when ordering is inconsistent (#3461) 2023-06-27 16:11:41 +02:00
Thomas
54d626d5cd
[XPathAbstract] Use baseURI to fix relative links (if available) (#3439) 2023-06-17 17:53:00 +02:00
Dag
0a8fe57003
feat: enable bridges using env var (#3428)
* refactor: bridgefactory, add tests

* refactor: move defaultly enabled bridges to config

* refactor

* refactor

* feat: add support for enabling bridges with env var
2023-06-11 03:16:03 +02:00
Dag
fbaf26e8bf
fix(html_format): add spacing below date if author is missing (#3425)
* small ui tweak

* remove unused <div>

* refactor: rename method

* refactor: inline const

* refactor
2023-06-08 23:04:16 +02:00
Dag
ee498eadf9
fix: move debug mode to config (#3324)
* fix: move debug mode to config

* fix: also move debug_whitelist to .ini config

* fix: move logic back to Debug class

* docs

* docs

* fix: disable debug mode by default

* fix: restore previous behavior for alerts

* fix: center-align alert text
2023-06-02 20:22:09 +02:00
mrnoname1000
87b9f2dd94
[core] Fix XPathAbstract while working around Simple HTML DOM bug (#3408) 2023-05-21 21:06:35 +02:00
Dag
f803ffa79a
fix: ArgumentCountError: DOMDocument::getElementsByTagName() expects exactly 1 argument, 2 given, #3406 (#3407) 2023-05-21 19:59:39 +02:00
mrnoname1000
3e0d024888
[core] Fix defaultLinkTo for simple_html_dom objects (#3404) 2023-05-20 00:02:17 +02:00
Dag
cfe81ab2ac
fix: Call to a member function setAttribute() on int, #3402 (#3403) 2023-05-19 16:05:52 +02:00
mrnoname1000
096c3bca73
[XPathAbstract] Fix relative links in fetched HTML (#3401)
* [core] Make defaultLinkTo compatible with DOMDocument

* [XPathAbstract] Fix relative links in fetched HTML
2023-05-18 13:50:50 +02:00
mrnoname1000
d0f7f5e2d8
[New Bridge] FiderBridge (#3378)
* [core] Add config parameter to markdownToHtml

* [FiderBridge] New bridge
2023-05-11 21:24:12 +02:00
Dag
50865d5741
fix: add additional cloudflare title (Glassdoor specific) (#3342) 2023-05-11 01:33:38 +02:00
mrnoname1000
c6c4b3a24f
[XPathAbstract] Fix encoding on feed title (#3365) 2023-05-11 01:32:01 +02:00
mrnoname1000
63dc500ae0
[XPathAbstract] Save HTML for entry content (#3366) 2023-05-11 01:31:34 +02:00
Dag
ff49c9f731
fix(TwitterBridge): repair fetching of tweets by username (#3385)
* feat: alpha version of new twitter bridge

* fix: refetch guest_token if expired

* fix: purge cache

* fix: safeguards

* fix

* fix: two notices

* fix

* fix: use factory to create cache

* fix: fail properly instead of die()
2023-05-10 21:45:44 +02:00
Max
c628f99928 use lowercase 2023-05-10 18:40:33 +02:00
Eugene Molotov
343fd36671
[core] Remove hardcoded maximum duration of 24 hours in loadCacheValue (#3355) 2023-04-19 17:53:35 +02:00
Dag
9bb04ba848
Prepare 2023-03-21 release (#3323)
* fix: upgrade version string in php code
2023-03-22 19:32:19 +01:00
Dag
4c3ebb312d
feat: improve error handling ux (#3298)
* feat: improve error handling ux

* feat: add error messages for failed xml parsing
2023-03-20 19:11:51 +01:00
Dag
9e9a697b8b
feat: add config option "path" for file cache (#3297) 2023-03-20 19:10:01 +01:00
Dag
058e792b8f
feat: add filecache config to enable/disable real purge (#3263)
* refactor: cachefactory

* feat: add filecache config to enable/disable real purge

* test: fix test
2023-03-06 21:50:40 +01:00
Dag
007f2b2d8a
feat: sanitize root folder also in php error messages (#3262) 2023-03-06 21:47:25 +01:00
Dag
a01c1f6ab0
fix: disallow usage of default password (#3284) 2023-03-06 20:43:44 +01:00
Bocki
f0e5ef0fc5
[Various] getKey replacements and docu (#3283)
* [Various] getKey replacements and docu

* more bridges and fix to the abstract

* linting

* revert bandcampdaily. doing more than i thought
2023-03-06 20:01:51 +01:00
Bocki
f3f98a117c
[Core] Add getKey function (#3275)
* [Core] Add getKey function
2023-03-02 13:25:57 +01:00
Mynacol
4450e9b973
Let curl select the default HTTP version (#3249)
This essentially reverts b042412416,
as YouTube seems to have fixed their servers.
At least I was able to query the YouTube endpoint around 150 times with
CURL_HTTP_VERSION_2TLS recently. They even advertise HTTP/3 support with
an `alt-svc` HTTP header now.

This unsets CURLOPT_HTTP_VERSION to let curl decide
on the version. This would support all curl versions and opens the
possibility for HTTP/3, but leads to inconsistent behavior depending
on the underlying curl version.

We don't set CURL_HTTP_VERSION_NONE explicitly, as it is always the curl
default and opens the path to let individual bridges override the HTTP
version where necessary.

Alternatively, setting CURL_HTTP_VERSION_2TLS explicitly would lead to consistent behavior
regardless of the curl version, but might uncover old curl bugs before the
developers enabled HTTP/2 by default.
Additionally, that requires at least PHP 7.0.7 (we require PHP 7.4
already) and curl 7.47.0 [1], released on Jan 27 2016 [2].

See also the discussion on https://github.com/RSS-Bridge/rss-bridge/pull/3249

[1] https://www.php.net/manual/curl.constants.php
[2] https://curl.se/docs/releases.html
2023-02-22 17:48:39 +01:00
Dag
c27a300e02
test: add failing mastodon test (#3255)
* fix: refactor cache factory

* test: add failing test

* add null cache
2023-02-15 21:22:37 +01:00
Dag
7f1b32f390
feat: add a proper feed item uid when the bridge errors out (#3237)
* refactor: move function to class

* fix: use the computed bridge name as cache key

* refactor: extract method

* fix: set a feed item uid on errors

* docs

* fix: remove year from uid
2023-02-02 22:53:01 +01:00
Dag
a13c4624fb
feat: custom http ua in AO3, fix #3188 (#3189)
* refactor

* feat: custom http ua in AO3, #3188
2022-12-13 09:53:42 +01:00
ORelio
d592e2cb15
[Core] Add html/convertLazyLoading (+ document stripRecursiveHTMLSection) (#3157)
* [core] Add html/convertLazyLoading($dom)

Looks for lazy-loading attributes such as 'data-src' and converts
them back to regular ones such as 'src', easier for RSS readers.
It also converts <picture> elements to plain <img> elements.

* [core] Document html/stripRecursiveHTMLSection()

Add documentation for that function (no code changes).

* [WordPressBridge] Use convertLazyLoading()

* [WordPressBridge] Unwrap image figures

<img> inside <figure> may not display on RSS readers.
This converts them back to <img>, without losing caption if present.

* [ZDNet] Convert lazy loading images

* [code] html/stripRecursiveHTMLSection: Fix typo
2022-11-20 12:41:59 +01:00
Eugene Molotov
8990aeb9f4
[core] Merge GET and POST args into request array (#3155) 2022-11-20 12:41:20 +01:00
Dag
88766e6fde
fix: produce smaller log records for http exceptions (#3153) 2022-11-18 21:36:06 +01:00
Dag
a5779d30b5
feat: add max file size to http responses (#3140) 2022-11-16 17:56:26 +01:00