* revamp http client to not limit requests, instead use sender worker
Signed-off-by: kim <grufwub@gmail.com>
* remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping
Signed-off-by: kim <grufwub@gmail.com>
* improve batch senders to keep popping recipients until remote URL found
Signed-off-by: kim <grufwub@gmail.com>
* fix recipient looping issue
Signed-off-by: kim <grufwub@gmail.com>
* fix missing mutex unlock
Signed-off-by: kim <grufwub@gmail.com>
* move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP
Signed-off-by: kim <grufwub@gmail.com>
* slight code reformatting
Signed-off-by: kim <grufwub@gmail.com>
* a whitespace
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code
Signed-off-by: kim <grufwub@gmail.com>
* add missing license headers
Signed-off-by: kim <grufwub@gmail.com>
* fix request backoff calculation
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* start working on notifs for new posts
* tidy up a bit
* update swagger
* carry over show reblogs + notify from follow req
* test notify on status post
* update column slice
* dedupe update logic + add tests
* fix own boosts not being timelined
* avoid type check, passing unnecessary accounts
* remove unnecessary 'inReplyToID' check
* add a couple todo's for future db functions
* start poking timelines
* OK yes we're refactoring, but it's nothing like the last time so don't worry
* more fiddling
* update tests, simplify Get
* thanks linter, you're the best, mwah mwah kisses
* do a bit more tidying up
* start buggering about with the prepare function
* fix little oopsie
* start merging lists into 1
* ik heb een heel zwaar leven
nee nee echt waar
* hey it works we did it reddit
* regenerate swagger docs
* tidy up a wee bit
* adjust paging
* fix little error, remove unused functions
* refactor visibility checking, add caching for visibility
* invalidate visibility cache items on account / status deletes
* fix requester ID passed to visibility cache nil ptr
* de-interface caches, fix home / public timeline caching + visibility
* finish adding code comments for visibility filter
* fix angry goconst linter warnings
* actually finish adding filter visibility code comments for timeline functions
* move home timeline status author check to after visibility
* remove now-unused code
* add more code comments
* add TODO code comment, update printed cache start names
* update printed cache names on stop
* start adding separate follow(request) delete db functions, add specific visibility cache tests
* add relationship type caching
* fix getting local account follows / followed-bys, other small codebase improvements
* simplify invalidation using cache hooks, add more GetAccountBy___() functions
* fix boosting to return 404 if not boostable but no error (to not leak status ID)
* remove dead code
* improved placement of cache invalidation
* update license headers
* add example follow, follow-request config entries
* add example visibility cache configuration to config file
* use specific PutFollowRequest() instead of just Put()
* add tests for all GetAccountBy()
* add GetBlockBy() tests
* update block to check primitive fields
* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests
* fix copy-pasted code
* update envparsing test
* whitespace
* fix bun struct tag
* add license header to gtscontext
* fix old license header
* improved error creation to not use fmt.Errorf() when not needed
* fix various rebase conflicts, fix account test
* remove commented-out code, fix-up mention caching
* fix mention select bun statement
* ensure mention target account populated, pass in context to customrenderer logging
* remove more uncommented code, fix typeutil test
* add statusfave database model caching
* add status fave cache configuration
* add status fave cache example config
* woops, catch missed error. nice catch linter!
* add back testrig panic on nil db
* update example configuration to match defaults, slight tweak to cache configuration defaults
* update envparsing test with new defaults
* fetch followingget to use the follow target account
* use accounnt.IsLocal() instead of empty domain check
* use constants for the cache visibility type check
* use bun.In() for notification type restriction in db query
* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())
* use bun query building for nested select statements to ensure working with postgres
* update public timeline future status checks to match visibility filter
* same as previous, for home timeline
* update public timeline tests to dynamically check for appropriate statuses
* migrate accounts to allow unique constraint on public_key
* provide minimal account with publicKey
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* start refactoring account deletion
* update to use state.DB
* further messing about
* some more tidying up
* more tidying, cleaning, nice-making
* further adventures in refactoring and the woes of technical debt
* update fr accept/reject
* poking + prodding
* fix up deleting
* create fave uri
* don't log using requestingAccount.ID because it might be nil
* move getBookmarks function
* use exists query to check for status bookmark
* use deletenotifications func
* fiddle
* delete follow request notif
* split up some db functions
* Fix possible nil pointer panic
* fix more possible nil pointers
* fix license headers
* warn when follow missing (target) account
* return wrapped err when bookmark/fave models can't be retrieved
* simplify self account delete
* warn log likely race condition
* de-sillify status delete loop
* move error check due north
* warn when unfollowSideEffects has no target account
* warn when no boost account is found
* warn + dump follow when no account
* more warnings
* warn on fave account not set
* move for loop inside anonymous function
* fix funky logic
* don't remove mutual account items on block;
do make sure unfollow occurs in both directions!
* [feature] Allow admins to send test emails
* implement unwrap on new error type
* add + use gtserror types
* GoToSocial Email Test -> GoToSocial Test Email
* add + use getInstance db call
* removed unused "unknown" error type
* [chore] Remove years from all license headers
Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.
In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.
* [chore] Add license header check
This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.
* [chore] Add missing license headers
* [chore] Further updates to license header
* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
* [feature] Provide .well-known/host-meta endpoint
This adds the host-meta endpoint as Mastodon clients use this to
discover the API domain to use when the host and account domains aren't
the same.
* Address review comments
* [feature] Discover webfinger through host-meta
This implements a fallback for discovering the webfinger endpoint in
case the /.well-known/webfinger endpoint wasn't properly redirected.
Some instances do this because the recommendation used to be to use
host-meta for the webfinger redirect in the before times.
Closes#1558.
* [bug] Ensure we only ever update cache on success
* [chore] Move finger tests to their own place
This adds a test suite for transport and moves the finger cache tests
into there instead of abusing the search test suite.
* [chore] cleanup the test a bit more
We don't really need a separate function for the oddly located webfinger
response as we check the full URL string anyway
* Address review comments
* [chore] update config example
* [chore] access DB only through state in controller
* Add go-playground/form pkg
* [feature] Add support for profile fields
* Add field attributes test
* Validate profile fields form
* Add profile field validation tests
* Add Field Attributes definition to swagger
---------
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
* Advertise rich text formats, support content_type field
* Update JSON in instance patch tests
* Replace format with content_type everywhere
* update migration to work with both pg and sqlite
* regenerate swagger docs
* update instance serialization + tests
* fix up
* learn to code tobi please, i'm begging you
---------
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* start fiddling
* the ol' fiddle + update
* start working on fetching statuses
* poopy doopy doo where r u uwu
* further adventures in featuring statuses
* finishing up
* fmt
* simply status unpin loop
* move empty featured check back to caller function
* remove unnecessary log.WithContext calls
* remove unnecessary IsIRI() checks
* add explanatory comment about status URIs
* change log level to error
* better test names
* simply use storage.Storage, removing wrapping KVStore as we don't need KV store locking functionality
Signed-off-by: kim <grufwub@gmail.com>
* fix missing unwrapped function
Signed-off-by: kim <grufwub@gmail.com>
* add code comment
Signed-off-by: kim <grufwub@gmail.com>
* linter, please take my offering in peace
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
This removes the current default of checking for membership of the admin
or admins group and makes it required to explicitly configure which
groups should grant admin access, if any.
Relying on the implicit default of admin or admins is potentially
dangerous as that group may contain a different subset of people that we
may wish to grant admin access to GtS. This is probably not an issue for
a single-person instance, but for a community instance different admin
groups may exist in an OIDC provider for different applications.
I'm explicitly opting for not defaulting the value of oidc-admin-groups
to admin,admins because I think it's better for those things to be
explicitly configured.
* implement status pin client api + web handler
* make test names + comments more descriptive
* don't use separate table for status pins
* remove unused add + remove checking
* tidy up + add some more tests
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.
Co-authored-by: kim <grufwub@gmail.com>
* [bugfix] set cache-control max-age dynamically for s3
* woops
* double whoops
* time until, thank you linter, bless you, you're the best, no matter what kim says
* aa
* serve publickey separately from AP, don't throttle it
* update nginx cache documentation, cache main-key too
* throttle public key, but separately from other endpoints
* remove local copying of file for satisfying range headers
Signed-off-by: kim <grufwub@gmail.com>
* remove unused import
Signed-off-by: kim <grufwub@gmail.com>
* fix range header reader limit calculation
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* reformat GetAccount() functionality, and add UpdateAccount() function.
* use fetched_at instead of last_webfingered_at
* catch local "not found" errors. small formatting / error string changes
* remove now unused error type
* return nil when wrapping nil error
* update expected error messages
* return correct url for foss satan webfinger
* add AP model for Some_User
* normalize local domain
* return notretrievable where appropriate
* expose NewErrNotRetrievable
* ensure webfinger for new accounts searched by uri
* update local account short circuit
* allow enrich to fail for already-known accounts
* remove unused LastWebfingeredAt
* expose test maps on mock http client
* update Update test
* reformat GetAccount() functionality, and add UpdateAccount() function.
* use fetched_at instead of last_webfingered_at
* catch local "not found" errors. small formatting / error string changes
* remove nil error checks (we shouldn't be passing nil errors to newError() initializers)
* remove mutex unlock on transport init fail (it hasn't yet been locked!)
* woops add back the error wrapping to use ErrNotRetrievable
* caches were never being started... 🙈
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* Implement goldmark debug print for hashtags and mentions
* Minify HTML in FromPlain
* Convert plaintext status parser to goldmark
* Move mention/tag/emoji finding logic into formatter
* Combine mention and hashtag boundary characters
* Normalize unicode when rendering hashtags
* start adding report client api
* route + test reports get
* start report create endpoint
* you can create reports now babyy
* stub account report processor
* add single reportGet endpoint
* fix test
* add more filtering params to /api/v1/reports GET
* update swagger
* use marshalIndent in tests
* add + test missing Link info
* don't serve unused fields for video attachments
* parse video bitrate + duration more accurately
* use ServeContent where appropriate to respect Range
* abstract temp file seeker into its own function
* media processor consolidation and reformatting, reduce amount of required syscalls
Signed-off-by: kim <grufwub@gmail.com>
* update go-store library, stream jpeg/png encoding + use buffer pools, improved media processing AlreadyExists error handling
Signed-off-by: kim <grufwub@gmail.com>
* fix duration not being set, fix mp4 test expecting error
Signed-off-by: kim <grufwub@gmail.com>
* fix test expecting media files with different extension
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code
Signed-off-by: kim <grufwub@gmail.com>
* fix expected storage paths in tests, update expected test thumbnails
Signed-off-by: kim <grufwub@gmail.com>
* remove dead code
Signed-off-by: kim <grufwub@gmail.com>
* fix cached presigned s3 url fetching
Signed-off-by: kim <grufwub@gmail.com>
* fix tests
Signed-off-by: kim <grufwub@gmail.com>
* fix test models
Signed-off-by: kim <grufwub@gmail.com>
* update media processing to use sync.Once{} for concurrency protection
Signed-off-by: kim <grufwub@gmail.com>
* shutup linter
Signed-off-by: kim <grufwub@gmail.com>
* fix passing in KVStore GetStream() as stream to PutStream()
Signed-off-by: kim <grufwub@gmail.com>
* fix unlocks of storage keys
Signed-off-by: kim <grufwub@gmail.com>
* whoops, return the error...
Signed-off-by: kim <grufwub@gmail.com>
* pour one out for tobi's code <3
Signed-off-by: kim <grufwub@gmail.com>
* add back the byte slurping code
Signed-off-by: kim <grufwub@gmail.com>
* check for both ErrUnexpectedEOF and EOF
Signed-off-by: kim <grufwub@gmail.com>
* add back links to file format header information
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* Add local user and post count to nodeinfo responses
This fixes#1307 (at least partially). The nodeinfo endpoint should now
return the total users on an instance, along with their post count.
* Update NodeInfoUsers docstring and swagger yaml file
* launch websocket streaming in goroutine to allow upgrade handler to return
* don't send any message on ping, improved close check on failed read
* use context to signal wsconn close, ensure canceled in read goroutine
Signed-off-by: kim <grufwub@gmail.com>
* interim commit: start refactoring middlewares into package under router
* another interim commit, this is becoming a big job
* another fucking massive interim commit
* refactor bookmarks to new style
* ambassador, wiz zeze commits you are spoiling uz
* she compiles, we're getting there
* we're just normal men; we're just innocent men
* apiutil
* whoopsie
* i'm glad noone reads commit msgs haha :blob_sweat:
* use that weirdo go-bytesize library for maxMultipartMemory
* fix media module paths
* close pipereader on failed data function
* gently slurp the bytes
* readability updates
* go fmt
* tidy up file server tests + add more cases
* start moving io wrappers to separate iotools package. Remove use of buffering while piping recache stream
Signed-off-by: kim <grufwub@gmail.com>
* add license text
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* [chore] Move ShortcodeDomain to its own little util func
* [feature] Add RefetchEmojis function to media manager
* [feature] Expose admin media refresh via admin API
* update following review feedback
- change/fix log levels
- make sure not to try to refetch local emojis
- small style refactoring + comments
* log on emoji refetch start
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* Implement Bookmarks
* Update based on review comments
* Update swagger doc
* Fix argument passing to status.Bookmark
* Update changed test
* Updates based on latest PR review
* [feature] overhaul the oidc system
this allows for more flexible username handling and prevents account
takeover using old email addresses
* [feature] add migration path for old OIDC users
* [feature] nicer error reporting for users
* [docs] document the new OIDC flow
* [fix] return early on oidc error
* [docs]: add comments on the finalization logic
* start refactoring some of the search + deref logic
* add tests for search api
* rename GetRemoteAccount + GetRemoteStatus
* make search function a bit simpler + clearer
* fix little fucky wucky uwu owo i'm just a little guy
* update faulty switch statements
* update test to use storage struct
* redo switches for clarity
* reduce repeated logic in search tests
* fastfail getstatus by uri
* debug log + trace log better
* add implementation note
* return early if no result for namestring search
* return + check on dereferencing error types
* errors hah what errors
* remove unneeded error type alias, add custom error text during stringification itself
* fix a woops recursion 🙈
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* start adding admin emoji PATCH stuff
* updating works OK, now how about copying
* allow emojis to be copied
* update swagger docs
* update admin processer to use non-interface storage driver
* remove shortcode updating for local emojis
* go fmt
Co-authored-by: f0x52 <f0x@cthu.lu>
* add account block DB cache and remove reliance on relational joins
* actually include cache key arguments...
* add a PutBlock() method which also updates the block cache, update tests accordingly
* use `PutBlock` instead of `Put(ctx, block)`
* add + use functions for deleting + invalidating blocks
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
Google abandoned Floc in favour of Topics and changed the
necessary permissions policy. Currently, the Floc policy will
block Topics. This change includes switching to the updated
policy to be ahead of Google abandoning recognising the Floc
policy.
Update the function documentation to include the current relevant
documentation.
* convert most of the caches to use result.Cache{}
* add caching of emojis
* fix issues causing failing tests
* update go-cache/v2 instances with v3
* fix getnotification
* add a note about the left-in StatusCreate comment
* update EmojiCategory db access to use new result.Cache{}
* fix possible panic in getstatusparents
* further proof that kim is not stinky
* [feature] Add 'role' field to api serialization of local accounts
* [chore] Add a bit of license text while I'm here
* [frogend] render account role on same line as username in web view of profile
* style tweaking on role badges, general profile header layout
* profile stats wrapping
* don't render standard 'user' role on web view
Co-authored-by: f0x <f0x@cthu.lu>
* [feature] Add emoji categories GET
Serialize emojis in appropriate categories; make it possible to get categories via the admin API
* [feature] Create (or use existing) category for new emoji uploads
* fix lint issue
* update misleading line in swagger docs
* Add instance-expose-public-timeline flag
Adds a config flag that allows unauthenticated access to /api/v1/timelines/public. Defaults to false to replicate existing behaviour.
* Update structure following review
* Add comment
* Fix linting
* Fix login on Mastodon iOS app for users with no statuses
Mastodon for iOS can't cope with an empty string for a date and
expect a JSON `null` instead.
Fixes https://github.com/superseriousbusiness/gotosocial/issues/1010
* Fix expected values in tests to match
* [feature] Make instance thumbnail configurable via admin panel
* log db errors in InstanceToAPIInstance
* only update instance in db if necessary
* start adding tests
* finish test
* re-add eslint
* fix oauth url getting too long
* actually attach single emoji get and delete routes
* basic emoji details + deletion using rtk query
* refactor emoji upload to rtk query
* clean up old redux api+reducers for custom emoji
* fix validation order
* refactor custom emoji form fields
* remove unused requires
* cleanup, fix most eslint errors
* more small eslint fixes
* fix max emoji size
* tiny bit of function documentation
* use readcloser for content.Content
* call media postdata function no matter what
* return a readcloser from data func
* tidy of logic of readertostore
* fix whoopsie
* select emoji using image_static_url
* use updated on AP emojis
* allow refetch of updated emojis
* cheeky workaround for test
* clean up old files for refreshed emoji
* check error for originalPostData
* shorten GetEmojiByStaticImageURL
* delete kirby (sorry nintendo)
* add admin emojis get path + model + docs
* stub admin emojis get processor function
* add id + disabled fields to admin emoji
* add emoji -> api admin emoji converter
* tidy up a bit
* add GetEmojis function
* finish up get emojis function
* order by shortcodedomain
* ASC
* tidy up + explain
* update to allow paging
* make admin emojis pageable
* fix mixed case paging
* normalize emoji queries a bit better
* test emoji get paging
* make limit optional
* fix incorrect path in media cleanup tests
* i have bad coder syndrome
* don't trimspace
* rename -> GetUseableEmojis
* wrap emoji query in subquery
avoid selecting more than we need
* fix a bit of sillyness teehee
* fix subquery postgres woes
* start adding rss functionality
* add gorilla/feeds dependency
* first bash at building rss feed
still needs work, this is an interim commit
* tidy up a bit
* add publicOnly option to GetAccountLastPosted
* implement rss endpoint
* fix test
* add initial user docs for rss
* update rss logo
* docs update
* add rssFeed to frontend
* feed -> feed.rss
* enableRSS
* increase rss logo size a lil bit
* add rss toggle
* move emojify to text package
* fiddle with rss feed formatting
* add Text field to test statuses
* move status to rss item to typeconverter
* update bun schema for enablerss
* simplify 304 checking
* assume account not rss
* update tests
* update swagger docs
* allow more characters in title, trim nicer
* update last posted to be more consistent
