Commit graph

3841 commits

Author SHA1 Message Date
sysadminstory
52d3cce59d
bridges: add context to detectParameters (#3607)
* bridges: add context to detectParameters

Some bridges did not return the context parameter but they used it in
the parameters

* bridges: add context to detectParameters

Fix test for InstagramBridge
2023-08-09 22:40:24 +02:00
ORelio
6cc4cf24dc
[FuturaSciences] Fix content extraction (#3487, #3488) (#3606) 2023-08-09 20:10:15 +02:00
sysadminstory
1fcf67f14a
[PepperBridgeAbstract] Fix deal origin (#3605)
Origin display has chenged : this commit follow the websites changes.

Fixes #3521
2023-08-09 17:36:02 +02:00
sysadminstory
f3896ed543
[ImgsedBridge] Add detectParameters feature to the bridge (#3604)
The bridge can detect the most common profile variation URL of
instagram.com or imgsed.com websites to extract the username.
2023-08-09 17:35:35 +02:00
ORelio
b86ee5778b
[SitemapBridge] Add SitemapBridge (#3602)
* [SitemapBridge] Add SitemapBridge

This bridge is a variant of CssSelectorBridge.
Instead of retrieving article list from home page,
retrieves article list from SEO sitemap.xml.
Requires CssSelectorBridge to be installed.

* [SitemapBridge] Code linting
2023-08-08 15:02:01 +02:00
adminvulcano
43ec82179b
[TldrTechBridge] Add Cybersecurity section (#3601) 2023-08-08 07:00:07 +02:00
Korytov Pavel
cf6d94dc2a
[EconomistBridge] Fix strange image urls (#3600) 2023-08-08 06:58:08 +02:00
Niehztog
3e3481bd7a
adds Nius bridge (#3599)
* adds Nius bridge

* fix linter errors

* fix linter errors

* fix linter errors

* fix extract author
2023-08-07 05:33:35 +02:00
User123698745
4976cd227e
[FeedExpander] support xhtml content / content with child elements (#3598)
* [core] support xhtml content type in FeedExpander

* [FilterBridge] change defaultValue to exampleValue

* [core] support content with child elements in FeedExpander
2023-08-04 22:14:08 +02:00
Tone
d32419ffcf
added the option for a sessioncookie in heiseBridge (#3596)
* added the option for a sessioncookie

with a valid cookie you can get full heise+ (paywall) articles

* formating

* lint

---------

Co-authored-by: Dag <me@dvikan.no>
2023-08-03 22:43:55 +02:00
User123698745
7661a78a43
[core] add bridge not found warning message to frontpage (#3591) 2023-08-03 03:10:24 +02:00
Dag
ed97ce8646
fix: dont fail for non-existing enabled bridge (#3589)
* fix: dont fail for non-existing enabled bridge

* yup
2023-08-01 19:35:15 +02:00
mruac
10f7b6f4f6
Fix php8.2 deprecated warning when using bridge specific configurations (#3587)
* Fix php8.2 deprecated warning

Fix php8.2 warning: `Deprecated: Creation of dynamic property is deprecated`

* fix

* refactor: remove unused method

---------

Co-authored-by: Dag <me@dvikan.no>
2023-08-01 19:35:06 +02:00
Dag
8e2353ad3e
fix: write to cache only if data is was not cached, fix #3586 (#3588) 2023-08-01 06:19:42 +02:00
Dag
7e4807530e
fix: various small fixes (#3580) 2023-07-31 20:43:18 +02:00
Dag
8b6eecea25
docs: add note about expensive operation (#3579) 2023-07-31 20:43:11 +02:00
ORelio
f8fd05f08f
[CssSelectorBridge] Handling of missing links (#3585)
When using parent element as URL selector:

* If no <a> inside some elements, ignore them
* If no <a> inside ALL elements, report an error

Fixes #3573 #issuecomment-1656943318
2023-07-31 19:07:34 +02:00
User123698745
f957eea300
[FallGuysBridge] new bridge (#3584) 2023-07-31 01:05:38 +02:00
User123698745
93eecdf79f
[core] fix new bridge PRs not generating html preview artifacts (#3583)
* [core] replace everything except bridge name to get a valid whitelist.txt

* [core] do not use hard code repository name to improve working with forks

* [core] trim bridge names from whitelist.txt to reduce chance of failure
2023-07-30 23:26:59 +02:00
mrtnvgr
3a57fc800b
DoujinStyleBridge: Update html tags (#3581) 2023-07-30 06:46:16 +02:00
Dag
701fe3cfed
fix: various small fixes (#3578) 2023-07-29 00:14:30 +02:00
Aaron F
11ce8b5dcd
CVEDetails got a new HTML layout. (#3577)
This fixes the parser for CVEDetails.
2023-07-27 23:54:17 +02:00
Korytov Pavel
f5f76f111b
[TldrTechBridge] Add Web Dev and Founders sections (#3576) 2023-07-26 22:59:49 +02:00
Korytov Pavel
bf4ea12719
[ScientificAmerican] Fix bridge (#3575) 2023-07-26 21:47:47 +02:00
ORelio
235c084820
[DilbertBridge] Remove bridge (#3574)
dilbert.com has closed down.
2023-07-26 20:41:48 +02:00
ORelio
977c0db382
[CssSelectorBridge] Improvements (#3537) (#3573)
* [CssSelectorBridge] Improvements (#3537)

* Improve parameter documentation / add tooltips
* Allow extracting content from home page instead of article page
* Keep titles from home page when every page <title> is the same

* [CssSelectorBridge] Code linting

* [CssSelectorBridge] Code linting (2)

* [CssSelectorBridge] Code linting (3)
2023-07-26 19:41:29 +02:00
csisoap
556bca58cf
[TwitterBridge] Fix search, user, list ID (#3566)
* Add ability to fetch user, list tweet

* Fix user, search, list ID although list still broke

* clear whitespace

* Revert CACHE_TIMEOUT

* clear whitespace, change single quote

* Clear PHP warning, add ability to get full-text if truncated

* Clear PHP warning

* clear warning

* clear whitespace

* Add check condition for mediaDetails.

* Add whitespace

* Add try catch exception for get full-text tweet

* clear warning

* clear warning
2023-07-25 22:36:41 +02:00
Dawid Wróbel
2cc89b767c
[AllegroBridge] fix non-functional bridge (#3571)
— fix cookie pattern
– use data analytics attributes wherever possible to avoid relying on obfuscated class names
— add support for promoted offers
— include sponsored and promoted offers by default
— some additional refactoring
2023-07-25 20:52:47 +02:00
Simon Alberny
1f6c2cd32c
Allocine Sorties movie date added (#3569) 2023-07-24 16:25:09 +02:00
Dag
b6fab20601
docs: improve readme (#3560)
* docs: improve readme
2023-07-23 23:05:56 +02:00
Dag
74635fd752
fix(DisplayAction): improve error handling and cache logic (#3558)
* fix(DisplayAction): improve error handling and cache logic

* restore prev timeouts

* refactor

* yup

* test: fix unit test

* leave twitter client unchanged

* leave twitter bridge unchanged
2023-07-23 23:05:35 +02:00
Eugene Molotov
38ca124de0
[VkBridge] Better title generation (#3563)
1. Use first parargraph only
2. Remove tags
3. Allow to use comma and colon in title
2023-07-22 14:00:12 +02:00
Dag
39a8346c53
fix(pokemonnews): throw if antibot, #3327 (#3562) 2023-07-21 20:52:20 +02:00
Dag
d08b2616ef
feat(twitter): use account icon as feed icon, fix #3348 (#3561) 2023-07-21 20:26:22 +02:00
Dag
0a118310cb
fix(sqlitecache): store blob as blob (#3555)
serialize() can return output with null bytes and other
non-text data. The prior behavior truncated data
which later results in unserialize() errors.

This happens when e.g. caching an object with a private field
or when caching e.g. a JPEG file (starts with 0xFFD8FFE1)

Fixes errors such as e.g.:

unserialize(): Error at offset 20 of 24 bytes at caches/SQLiteCache.php line 51
2023-07-20 19:11:13 +02:00
Predä
663729cf19
[TikTokBridge] Use another way to get videos infos to include video link (#3557)
* [TikTokBridge] Use another way to get videos infos to include video link

* [TikTokBridge] Use cover if dynamicCover is empty

* [TikTokBridge] Add support for the rest of item params
2023-07-20 05:50:45 +02:00
Predä
2ffb54c7c2
[PicukiBridge] Add count parameter (#3556) 2023-07-20 00:52:09 +02:00
Dag
517c7f5c9b
fix(cache): bug (#3554) 2023-07-19 22:18:42 +02:00
Dag
93620aa105
fix(cache): bug in cache logic (#3553)
It is possible to have a cached item with a very old mtime but it's technically expired.

So, check for presence of time and whether the time it is within 10 days
2023-07-19 22:05:26 +02:00
Dag
a4a328583a
fix(reddit): set custom http ua to fix 429 errors (#3552)
* refactor

* refactor

* fix(reddit): set custom http ua to fix 429 errors

* lint
2023-07-19 06:39:17 +02:00
Dag
f91723d9e5
fix(memcached): do not flush entire cache, oops (#3551) 2023-07-19 05:18:26 +02:00
Dag
6254b8593e
refactor(cache): extract and encapsulate cache expiration logic (#3547)
* refactor(cache): extract and encapsulate cache expiration logic

* fix: logic bug in getSimpleHTMLDOMCached

* fix: silly me, index should of course be on the key column

* silly me again, PRIMARY keys get index by default lol

* comment out the delete portion in loadData

* remove a few log statements

* tweak twitter cache timeout
2023-07-19 05:05:49 +02:00
sysadminstory
087e790ec1
[ImgsedBridge] Add new Instagram Bridge Alternative (#3550)
* [ImgsedBridge] Add new Instagram Bridge Alternative

Imgsed is a Website adverstised on instagram website, that's is not
behind Cloudflare Anti Bot feature.
You can select to display Posts, Tags, and Stories of a specific
username

* [ImgsedBridge] Fix empty defaultValue
2023-07-19 03:28:14 +02:00
mrtnvgr
4ce63c88aa
Add DoujinStyleBridge (#3549)
* v1

* improve title

* search support

* random support

* fix categories

* add metadata to content

* fix linter errors

* i'm sorry
2023-07-18 20:48:29 +02:00
Paroleen
a1bae7a9a8
[SpotifyBridge] Add search API support (#3548) 2023-07-18 00:43:08 +02:00
Dag
08d16322e1
fix: bug in prior conflict merge (#3546) 2023-07-16 22:37:37 +02:00
Dag
440adf2f3b
fix(githubissue): add 10 min cache (#3545) 2023-07-16 22:28:20 +02:00
Dag
a59793e8d6
refactor: extract CurlHttpClient (#3532)
* refactor: extract CurlHttpClient

* refactor

* interface
2023-07-16 22:07:34 +02:00
Dag
7b46b97abd
refactor(spotify): replace manual curl with getContents (#3544) 2023-07-16 21:50:44 +02:00
Dag
310160fd92
feat: improve http 429 handling (#3541) 2023-07-16 07:18:38 +02:00