owncast/utils/utils.go
Gabe Kangas b6f68628c0
Chat refactor + persistent backing chat users (#1163)
* First pass at chat user registration and validation

* Disable chat if the user is disabled/blocked or the server hits max connections

* Handle dropping sockets if chat is disabled

* Fix origin in automated chat test

* Work for updated chat moderation

* Chat message markdown rendering and fix tests

* Put /api/chat behind a chat user access token. Closes #1085

* Reject blocked username changes

* More WIP moderation

* Defer configuring chat until we know if it is enabled. Closes #1135

* chat user blocking. Closes #1096

* Add tests around user access for #1096

* Add external integration chat message API + update integration auth middleware to pass along integration name. Closes #1092

* Delete old chat messages from db as to not hold on to excessive data. Closes #1152

* Add schema migration for messages. Closes #1155

* Commit updated API documentation

* Add chat load test

* Shared db mutex and db optimizations

* Simplify past display name handling

* Use a new test db for each test run

* Wire up the external messages actions + add tests for them

* Move access tokens to be actual users

* Run message pruning at launch + fix comparison

* Do not return API users in disabled users response

* Fix incorrect highlighting. Closes #1160

* Consolidate user table statements

* Set the max process connection limit to 70% of maximum

* Fix wrong old display name being returned in name change event

* Delete the old chat server files

* Wire back up the webhooks

* Remove unused

* Invalidate user cache on changes

* Do not send rendered body as RawBody

* Some cleanup

* Standardize names for external API users to ExternalAPIUser

* Do not log token

* Checkout branch when building admin for testing

* Bundle in dev admin for testing

* Some cleanup

* Cleanup js logs

* Cleanup and standardize event names

* Clean up some logging

* Update API spec. Closes #1133

* Commit updated API documentation

* Change paths to be better named

* Commit updated API documentation

* Update admin bundle

* Fix duplicate event name

* Rename scope var

* Update admin bundle

* Move connected clients controller into admin package

* Fix collecting usernames for autocomplete purposes

* No longer generate username when it is empty

* Sort clients and users by timestamp

* Move file to admin controller package

* Swap, so the comments stay correct

Co-authored-by: Jannik <jannik@outlook.com>

* Use explicit type alias

Co-authored-by: Jannik <jannik@outlook.com>

* Remove commented code.

Co-authored-by: Jannik <jannik@outlook.com>

* Cleanup test

* Remove some extra logging

* Add some clarity

* Update dev instance of admin for testing

* Consolidate lines

Co-authored-by: Jannik <jannik@outlook.com>

* Remove commented unused vars

Co-authored-by: Jannik <jannik@outlook.com>

* Until needed do not return IP address with client list

* Fix typo of wrong var

* Typo led to a bad test. Fix typo and fix test.

* Guard against the socket reconnecting on error if previously set to shutdown

* Do not log access tokens

* Return success message on enable/disable user

* Clean up some inactionable error messages. Sent ban message. Sort banned users.

* fix styling for when chat is completely disabled

* Unused

* guard against nil clients

* Update dev admin bundle

* Do not unhide messages when unblocking user just to be safe. Send removal action from the controller

* Add convinience function for getting active connections for a single user

* Lock db on these mutations

* Cleanup force disconnect using GetClientsForUser and capture client reference explicitly

* No longer re-showing banned user messages for safety. Removing this test.

* Remove no longer needed comment

* Tweaks to forbidden username handling.

- Standardize naming to not use "block" but "forbidden" instead.
- Pass array over the wire instead of string.
- Add API test
- Fix default list incorrectly being appended to custom list.

* Logging cleanup

* Update dev admin bundle

* Add an artificial delay in order to visually see message being hidden when testing

* Remove the user cache as it is a premature optimization

* When connected to chat let the user know their current user details to sync the username in the UI

* On connected send current display name back to client.
- Move name change out of chat component.
- Add additional event type constants.

* Fix broken workflow due to typo

* Troubleshoot workflow

* Bump htm from 3.0.4 to 3.1.0 in /build/javascript (#1181)

* Bump htm from 3.0.4 to 3.1.0 in /build/javascript

Bumps [htm](https://github.com/developit/htm) from 3.0.4 to 3.1.0.
- [Release notes](https://github.com/developit/htm/releases)
- [Commits](https://github.com/developit/htm/compare/3.0.4...3.1.0)

---
updated-dependencies:
- dependency-name: htm
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Run npm run build and update libraries

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Gabe Kangas <gabek@real-ity.com>

* Commit updated Javascript packages

* Re-send current user info when a rejected name change takes place

* All socket writes should be through the send chan and not directly

* Seed the random generator

* Add keys and indexes to users table

* a util to generate consistent emoji markup

* console clean up

* mod tidy

* Commit updated API documentation

* Handle the max payload size of a socket message.
- Only close socket if x2 greater than the max size.
- Send the user a message if a message is too large.
- Surface the max size in bytes in the config.

* Update admin bundle

* Force all events to be sent in their own socket message and do not concatinate in a single message

* Update chat embed to register for access token

* Use a different access token for embed chat

* Update the chat message bubble background color to be bolder

* add base tag to open links in new window, closes #1220

* Support text input of :emoji: in chat (#1190)

* Initial implementation of emoji injection

* fix bookkeeping with multiple emoji

* make the emoji lookup case-insensitive

* try another solution for Caretposition

* add title to emojis

minor refactoring

* bind moji injection to InputKeyUp

* simplify the code

replace all found emojis

* inject emoji if the modifer is released earlier

* more efficient emoji tag search

* use json emoji.emoji as url

* use createEmojiMarkup()

* move emojify() to chat.js

* emojify on paste

* cleanup emoji titles in paste

* update inputText in InputKeyup

* mark emoji titles with 2*zwnj

this way paste cleanup will not interfere with text which include zwnj

* emoji should not change the inputText

* Do not show join messages when chat is offline. Closes #1224
- Show stream starting/ending messages in chat.
- When stream starts show everyone the welcome message.

* Force scrolling chat to bottom after history is populated regardless of scroll position. Closes https://github.com/owncast/owncast/issues/1222

* use maxSocketPayloadSize to calculate total bytes of message payload (#1221)

* utilize maxSocketPayloadSize from config; update chatInput to calculate based on that value instead of text value; remove usage of inputText for counting

* add a buffer to account for entire websocket payload for message char counting; trim nbsp;'s from ends of messages when calculating count

Co-authored-by: Gabe Kangas <gabek@real-ity.com>

Co-authored-by: Owncast <owncast@owncast.online>
Co-authored-by: Jannik <jannik@outlook.com>
Co-authored-by: Ginger Wong <omqmail@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Meisam <39205857+MFTabriz@users.noreply.github.com>
2021-07-19 19:22:29 -07:00

257 lines
6.1 KiB
Go

package utils
import (
"bytes"
"errors"
"fmt"
"io/ioutil"
"math/rand"
"net/url"
"os"
"os/exec"
"path"
"path/filepath"
"strings"
"github.com/mssola/user_agent"
log "github.com/sirupsen/logrus"
"github.com/yuin/goldmark"
"github.com/yuin/goldmark/extension"
"github.com/yuin/goldmark/renderer/html"
"mvdan.cc/xurls"
)
// DoesFileExists checks if the file exists.
func DoesFileExists(name string) bool {
if _, err := os.Stat(name); err == nil {
return true
} else if os.IsNotExist(err) {
return false
} else {
log.Errorln(err)
return false
}
}
// GetRelativePathFromAbsolutePath gets the relative path from the provided absolute path.
func GetRelativePathFromAbsolutePath(path string) string {
pathComponents := strings.Split(path, "/")
variant := pathComponents[len(pathComponents)-2]
file := pathComponents[len(pathComponents)-1]
return filepath.Join(variant, file)
}
func GetIndexFromFilePath(path string) string {
pathComponents := strings.Split(path, "/")
variant := pathComponents[len(pathComponents)-2]
return variant
}
// Copy copies the file to destination.
func Copy(source, destination string) error {
input, err := ioutil.ReadFile(source)
if err != nil {
return err
}
return ioutil.WriteFile(destination, input, 0600)
}
// Move moves the file to destination.
func Move(source, destination string) error {
return os.Rename(source, destination)
}
// IsUserAgentABot returns if a web client user-agent is seen as a bot.
func IsUserAgentABot(userAgent string) bool {
if userAgent == "" {
return false
}
botStrings := []string{
"mastodon",
"pleroma",
"applebot",
}
for _, botString := range botStrings {
if strings.Contains(strings.ToLower(userAgent), botString) {
return true
}
}
ua := user_agent.New(userAgent)
return ua.Bot()
}
func RenderSimpleMarkdown(raw string) string {
markdown := goldmark.New(
goldmark.WithRendererOptions(
html.WithUnsafe(),
),
goldmark.WithExtensions(
extension.NewLinkify(
extension.WithLinkifyAllowedProtocols([][]byte{
[]byte("http:"),
[]byte("https:"),
}),
extension.WithLinkifyURLRegexp(
xurls.Strict,
),
),
),
)
trimmed := strings.TrimSpace(raw)
var buf bytes.Buffer
if err := markdown.Convert([]byte(trimmed), &buf); err != nil {
panic(err)
}
return buf.String()
}
func RenderPageContentMarkdown(raw string) string {
markdown := goldmark.New(
goldmark.WithRendererOptions(
html.WithUnsafe(),
),
goldmark.WithExtensions(
extension.GFM,
extension.NewLinkify(
extension.WithLinkifyAllowedProtocols([][]byte{
[]byte("http:"),
[]byte("https:"),
}),
extension.WithLinkifyURLRegexp(
xurls.Strict,
),
),
),
)
trimmed := strings.TrimSpace(raw)
var buf bytes.Buffer
if err := markdown.Convert([]byte(trimmed), &buf); err != nil {
panic(err)
}
return strings.TrimSpace(buf.String())
}
// GetCacheDurationSecondsForPath will return the number of seconds to cache an item.
func GetCacheDurationSecondsForPath(filePath string) int {
if path.Base(filePath) == "thumbnail.jpg" {
// Thumbnails re-generate during live
return 20
} else if path.Ext(filePath) == ".js" || path.Ext(filePath) == ".css" {
// Cache javascript & CSS
return 60
} else if path.Ext(filePath) == ".ts" {
// Cache video segments as long as you want. They can't change.
// This matters most for local hosting of segments for recordings
// and not for live or 3rd party storage.
return 31557600
} else if path.Ext(filePath) == ".m3u8" {
return 0
}
// Default cache length in seconds
return 30
}
func IsValidUrl(urlToTest string) bool {
if _, err := url.ParseRequestURI(urlToTest); err != nil {
return false
}
u, err := url.Parse(urlToTest)
if err != nil || u.Scheme == "" || u.Host == "" {
return false
}
return true
}
// ValidatedFfmpegPath will take a proposed path to ffmpeg and return a validated path.
func ValidatedFfmpegPath(ffmpegPath string) string {
if ffmpegPath != "" {
if err := VerifyFFMpegPath(ffmpegPath); err == nil {
return ffmpegPath
} else {
log.Warnln(ffmpegPath, "is an invalid path to ffmpeg will try to use a copy in your path, if possible")
}
}
// First look to see if ffmpeg is in the current working directory
localCopy := "./ffmpeg"
hasLocalCopyError := VerifyFFMpegPath(localCopy)
if hasLocalCopyError == nil {
// No error, so all is good. Use the local copy.
return localCopy
}
cmd := exec.Command("which", "ffmpeg")
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatalln("Unable to determine path to ffmpeg. Please specify it in the config file.")
}
path := strings.TrimSpace(string(out))
return path
}
// VerifyFFMpegPath verifies that the path exists, is a file, and is executable.
func VerifyFFMpegPath(path string) error {
stat, err := os.Stat(path)
if os.IsNotExist(err) {
return errors.New("ffmpeg path does not exist")
}
if err != nil {
return fmt.Errorf("error while verifying the ffmpeg path: %s", err.Error())
}
if stat.IsDir() {
return errors.New("ffmpeg path can not be a folder")
}
mode := stat.Mode()
//source: https://stackoverflow.com/a/60128480
if mode&0111 == 0 {
return errors.New("ffmpeg path is not executable")
}
return nil
}
// Removes the directory and makes it again. Throws fatal error on failure.
func CleanupDirectory(path string) {
log.Traceln("Cleaning", path)
if err := os.RemoveAll(path); err != nil {
log.Fatalln("Unable to remove directory. Please check the ownership and permissions", err)
}
if err := os.MkdirAll(path, 0777); err != nil {
log.Fatalln("Unable to create directory. Please check the ownership and permissions", err)
}
}
func FindInSlice(slice []string, val string) (int, bool) {
for i, item := range slice {
if item == val {
return i, true
}
}
return -1, false
}
// GenerateRandomDisplayColor will return a random _hue_ to be used when displaying a user.
// The UI should determine the right saturation and lightness in order to make it look right.
func GenerateRandomDisplayColor() int {
rangeLower := 0
rangeUpper := 360
return rangeLower + rand.Intn(rangeUpper-rangeLower+1) //nolint
}