Spacebar/docs
1
0
Fork 0
mirror of https://github.com/spacebarchat/docs.git synced 2024-11-22 10:12:31 +01:00
Code Issues Releases Activity

Add built-in image proxy & misc stuff

Browse Source
This commit is contained in:
TomatoCake 2024-08-25 15:10:02 +02:00
parent 2adfa02f0d
commit 847a7f8c0c
17 changed files with 149 additions and 144 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/assets/js/missingroutes.js
View File

@ -19,7 +19,7 @@ document
const json = await res.json();
const missingRoutes = json.routes;
document.getElementById("counter").innerHTML =
document.getElementById("counter").textContent =
`We implement ${json.discord - json.missing}/${
json.discord
} endpoints from Discord.com ` +

8
docs/contributing/index.md
View File

@ -16,7 +16,7 @@
## Structure
{{ project.name }} is written in Typescript and is comprised of 4 main parts:
{{ project.name }} is written in TypeScript and is comprised of 4 main parts:
- REST HTTP API server
- Websocket Gateway server for realtime communication with clients
@ -29,7 +29,7 @@ Generally, the approach is to just see what the Discord.com client sends and rec
and guessing about any functionality server-side, if it's undocumented.
For a lot of things it's pretty simple to guess, `GET /api/users/@me` returns private details about your user for example.
This route is also detailed in [Discords own documentation](https://discord.com/developers/), [here specifically](https://discord.com/developers/docs/resources/user#get-current-user).
This route is also detailed in [Discords own documentation](https://discord.com/developers/docs/intro), [here specifically](https://discord.com/developers/docs/resources/user#get-current-user).
Discord generally does not document anything that is not related to application/bot development, though.
As an example, `GET /api/updates?platform={}` which returns the `url`, `pub_date`, `name` and any `notes` about the latest client release for a platform.
@ -40,5 +40,5 @@ Easy fix though, just edit the `DeveloperOptionsStore` localStorage key so that
!!! warning
Make sure you rerun `npm run build` every time you edit source code. Additionally, make sure you run `npm run generate:schema` whenever you change a
schema. If you want to do both, there's a shortcut: `npm run setup`.
Make sure you rerun `npm run build` every time you edit source code, or just use `npm run watch` to make TypeScript automatically recompile on code changes.
Wenn making changes to schemas or HTTP routes, run `npm run generate:schemas` and `npm run generate:openapi` to update the schemas used for validating incoming requests and generating the API documentation.

10
docs/contributing/instances.md
View File

@ -12,15 +12,15 @@ Your instance:
5. Must have at least regular uptime, meaning it is available at a consistent time of day.
6. Must have a valid and monitored [`general_correspondenceEmail` config](/setup/server/configuration) set.
7. Must not have default [rights](/setup/server/security/rights) that include operator or other administrative rights.
8. Enable [Imagor](/setup/server/configuration/imagor), as no image proxy allows attackers to learn user IP addresses.
8. Use an [image proxy](/setup/server/configuration/imageProxy), e.g. Imagor, as no image proxy allows attackers to learn user IP addresses.
9. Have a valid SSL/TLS certificate for all endpoints.
We recommend (not required) that you:
- Enable [Email verification](/setup/server/email), for anti-spam purposes
- Enable [Captcha](/setup/server/security/captcha), for anti-spam purposes
- Run your instance under [SystemD](/setup/server/systemd) or a similar system in your distro, for automatic restarting
- Provide some mechanism for users to report content. This may be as simple as more openly advertising your correspondence email (i.e. outside `GET /api/policies/instance` or `/api/ping`)
- Enable [Email verification](/setup/server/email), for anti-spam purposes.
- Enable [Captcha](/setup/server/security/captcha), for anti-spam purposes.
- Run your instance under [SystemD](/setup/server/systemd) or a similar system in your distro, for automatic restarting.
- Provide some mechanism for users to report content. This may be as simple as more openly advertising your correspondence email (i.e. outside `GET /api/policies/instance` or `/api/ping`).
- Provide some mechanism for instance status, such as [Grafana](https://grafana.com/).
- Host a [`/.well-known/spacebar`](/setup/server/wellknown) file on the domain you wish users associate with your instance, e.g. `spacebar.chat`.
If doing so, use this domain as the `url` field in your community instances PR.

6
docs/contributing/server/missingroute.md
View File

@ -1,11 +1,11 @@
# Missing Routes
Below is a list of routes available on Discord.com {{ project.name }}-server does not currently implement.
Below is a list of routes available on Discord.com that the {{ project.name }} server does not currently implement.
It does not account for different HTTP methods (GET, POST, etc). A single method implemented by {{ project.name }} will remove it from this list,
It does not account for different HTTP methods (GET, POST, etc). A single method implemented will remove it from this list,
so be sure to double check in the {{ project.name }} [source code]({{ repositories.base_url }}/{{ repositories.server }}/tree/master/src/api/routes).
It is generated daily by [{{ repositories.missing_routes }}]({{ repositories.base_url }}/{{ repositories.missing_routes }}/),
It is generated automatically by [{{ repositories.missing_routes }}]({{ repositories.base_url }}/{{ repositories.missing_routes }}/),
by scraping the latest Discord.com client.
<div>

16
docs/setup/bots/index.md
View File

@ -13,10 +13,10 @@ with the appropriate URLs of the instance you want to connect to.
You can get them from a client or from the [well-known](server/wellknown) instance endpoint.
### Discord.js
### discord.js
The `Client` class constructor accepts a `http` object, which you can use to change
the endpoints used.
The `Client` class constructor accepts configuration options that can be used to change
the endpoints.
```js
const { Client } = require("discord.js");
@ -36,7 +36,7 @@ const client = new Client({
client.login("your token here");
```
### Discord.py
### discord.py
```py
import discord
@ -57,13 +57,13 @@ client.run("your token here")
@NotNull
@Override
public String getGateway() {
return "wss://{REPLACE HERE WITH YOUR GATEWAY SERVER URL}/?encoding=json&v=9&compress=zlib-stream";
return "wss://gateway.{{ project.domain }}/?encoding=json&v=9&compress=zlib-stream";
}
```
5. Finally, configure JDA to use your RestConfig & SpacebarSessionController, like this:
```java
JDA jda = JDABuilder.createDefault("your token here")
.setRestConfig(restConfig)
.setSessionController(new SpacebarSessionController())
.build();
.setRestConfig(restConfig)
.setSessionController(new SpacebarSessionController())
.build();
```

5
docs/setup/server/configuration/embeds.md
View File

@ -4,7 +4,7 @@ Embeds in {{ project.name }} are external content that is displayed within your
{{ project.name }} tries its best to fetch content from these external sites,
but sometimes we require an API key or other authentication with the site to display their content (or nicer looking embeds).
For external images, it's best to set up [Imagor](imagor.md) for image resizing.
For external images, it's best to set up [Imagor or another image proxy](imageProxy.md) for image resizing.
The client may or may not fetch images directly from their source if this is not set up,
and as such some users may not see all images.
In the case where a client does fetch an image from it's source, without Imagor an attacker
@ -15,5 +15,6 @@ may be able to learn the IP addresses of users.
Go to the [Twitter developer portal](https://developer.twitter.com/) and sign up for developer access.
You must have a phone number attached to your account to sign up.
Create an application for use with the Twitter API (We don't need to post messages to Twitter itself).
Make sure to create a bearer token for authentication.
Make sure to create a Bearer token for authentication.
Once you have your API key, set the `external_twitter` [config](index.md) value to your API key, wrapped in quotes.

36
docs/setup/server/configuration/env.md
View File

@ -4,21 +4,21 @@ Below is a list of environment variables used by {{ project.name }}.
You can set environment variables easily by creating an `.env` file
in the `{{ project.name.lower() }}-server` folder, with the format `NAME=VALUE` with each on new lines.
| Name | Value | Description |
| ---------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------- |
| THREADS | number | Number of threads to run {{ project.name }} on when using bundle. Make sure you've enabled RabbitMQ if using more than one |
| PORT | number | Port to listen on. Used by all components, including bundle. If using bundle, all components run under the same port |
| DATABASE | string | Database connection string. Defaults to SQlite3 at project root |
| DB_SYNC | bool | Unsafely synchronise database model on startup. ***Not recommended for use in production environments!*** |
| CONFIG_PATH | string | File path for JSON config, if not using `config` db table |
| WS_LOGEVENTS | boolean | If set, log websocket events except messages from gateway |
| WS_VERBOSE | boolean | If set, log websocket messages sent/received by gateway |
| WS_DUMP | boolean | If set, dump websocket messages sent/received to disk |
| CDN | string | Lowest priority value for public CDN annoucements |
| GATEWAY | string | Lowest priority value for public gateway annoucements |
| STORAGE_LOCATION | string | CDN storage location. File path or S3 bucktet |
| STORAGE_PROVIDER | "s3" or "file" | CDN storage provider |
| STORAGE_BUCKET | string | S3 bucket name |
| STORAGE_REGION | string | S3 storage region |
| DB_LOGGING | boolean | if "true" logs all SQL queries to the terminal |
| LOG_REQUESTS | filter | What requests to log, per response code (eg. `-200` to log every non-200 response code, or `404` to log requests with a not found status code) |
| Name | Value | Description |
| ---------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| THREADS | number | Number of threads to run {{ project.name }} on when using bundle. Make sure you've enabled RabbitMQ if using more than one |
| PORT | number | Port to listen on. Used by all components, including bundle. If using bundle, all components run under the same port |
| DATABASE | string | Database connection string. Defaults to SQLite3 at project root |
| DB_SYNC | bool | Unsafely synchronise database model on startup. **Not recommended for use in production environments!** |
| CONFIG_PATH | string | File path for JSON config, if not using `config` db table |
| WS_LOGEVENTS | boolean | If set, log websocket events except messages from gateway |
| WS_VERBOSE | boolean | If set, log websocket messages sent/received by gateway |
| WS_DUMP | boolean | If set, dump websocket messages sent/received to disk |
| CDN | string | Lowest priority value for public CDN announcements |
| GATEWAY | string | Lowest priority value for public gateway announcements |
| STORAGE_LOCATION | string | CDN storage location. File path or S3 bucket |
| STORAGE_PROVIDER | "s3" or "file" | CDN storage provider |
| STORAGE_BUCKET | string | S3 bucket name |
| STORAGE_REGION | string | S3 storage region |
| DB_LOGGING | boolean | If "true", logs all SQL queries to the terminal |
| LOG_REQUESTS | filter | What requests to log, per response code (eg. `-200` to log every non-200 response code, or `404` to log requests with a Not Found status code) |

94
docs/setup/server/configuration/imageProxy.md Normal file
View File

@ -0,0 +1,94 @@
# Image proxy
If the configuration `cdn_imagorServerUrl` is empty (which is the default), `proxy_url` is set to the same value as the original `url`, causing clients to download images directly from the image host.
This may lead to privacy concerns, as an attacker is able to learn users IP addresses.
To combat this, {{ project.name }} supports using an image proxy to serve and resize images.
## Imagor
[Imagor](https://github.com/cshum/imagor) is a "fast, secure image processing server"
used by {{ project.name }} for external image resizing, primarily by embeds from other websites when linked in a message.
### Dependencies
- [Docker](https://www.docker.com/)
### Setup
To setup Imagor for {{ project.name }}, first grab the `security_requestSignature` config value from {{ project.name }}'s database,
and create a `imagor.env` file somewhere safe, with the following content.
**Make sure to edit the file with the correct information**. Your requestSignture should _not_ start or end with `"`.
```
IMAGOR_SECRET=security_requestSignature value from your {{ project.name }} config
PORT=8000
```
You can now start Imagor with
```bash
docker run --env-file ./imagor.env -p 8000:8000 shumc/imagor
```
`8000` here is our port. Make sure that it'd available to people outside your network.
If you would like to change the port Imagor listens on, be sure to change both the PORT value in `imagor.env`,
and the `-p` value used in docker.
If you're using a [reverse proxy](../reverseProxy.md) such as Nginx for {{ project.name }} already, you could add this to your config's `server` block
```nginx
location /media/ {
# If you changed the port, be sure to change it here too
proxy_pass http://127.0.0.1:8000/;
}
```
Along with any additional config you already have, of course.
Alternative (and perhaps the better choice) would be to create a new domain, say `media.example.com` specifically for Imagor.
```nginx
server {
# Change the server_name to reflect your true domain
server_name media.example.com;
add_header Last-Modified $date_gmt;
proxy_set_header Host $host;
proxy_pass_request_headers on;
proxy_set_header X-Forwarded-For $remote_addr;
location / {
# If you had changed the port, change it here as well
proxy_pass http://127.0.0.1:8000;
}
}
```
Our last step is to simply tell {{ project.name }} about Imagor. Just set the `cdn_imagorServerUrl` config value to your public endpoint for Imagor, wrapped in quotes.
For example, if you used the `/media` location in your existing nginx config, it will look something like `"https://{{ project.domain }}/media"`.
If you used a subdomain, it will look like `"https://media.{{ project.domain }}"`.
Don't include a trailing backslash.
Congrats! After a restart, you've now got Imagor resizing your images!
## Built-in image proxy
While we recommend using Imagor, {{ project.name }} has a built-in image proxy that can be used instead of Imagor if you just want to test the server or are unable to use Docker.
To use it, simply set the `cdn_imagorServerUrl` to your API URL, but use `/imageproxy` instead of `/api`.
For example, if the API is running at `https://api.{{ project.domain }}/api`, you would set `cdn_imagorServerUrl` to `"https://api.{{ project.domain }}/imageproxy"`.
Note that to support resizing the image to the given format, it's required to install either [sharp](https://www.npmjs.com/package/sharp) (preferred) or [jimp](https://www.npmjs.com/package/jimp):
```bash
npm install sharp
```
## Your own image proxy
You can also point the `cdn_imagorServerUrl` [config](index.md) to your own image proxy, if you have one that supports the Imagor URL format:
1. Construct URL path: `<Image width>x<Height>/<Unencoded URL without https://>`
2. Create a SHA-1 HMAC of the URL path using `security_requestSignature`
3. Construct final `proxy_url`: `<cdn_imagorServerUrl configuration>/<SHA-1 hash>/<Path from above>`

71
docs/setup/server/configuration/imagor.md
View File

@ -1,71 +0,0 @@
# Imagor
[Imagor](https://github.com/cshum/imagor) is a "fast, secure image processing server"
used by {{ project.name }} for external image resizing, primarily by embeds from other websites when linked in a message.
If left unused, {{ project.name }} will simply not provide a proxy_url for clients, which may leave external images unavailable
or cause the client to download directly from the image host. Downloading images directly from the host may lead to
privacy concerns, as an attacker may be able to learn users IP addresses.
## Dependencies
- [Docker](https://www.docker.com/)
## Setup
To setup Imagor for {{ project.name }}, first grab the `security_requestSignature` config value from {{ project.name }}'s database,
and create a `imagor.env` file somewhere safe, with the following content.
**Make sure to edit the file with the correct information**. Your requestSignture should _not_ start or end with `"`.
```
IMAGOR_SECRET=security_requestSignature value from your {{ project.name }} config
PORT=8000
```
You can now start Imagor with
```bash
docker run --env-file ./imagor.env -p 8000:8000 shumc/imagor
```
`8000` here is our port. Make sure that it'd available to people outside your network.
If you would like to change the port Imagor listens on, be sure to change both the PORT value in `imagor.env`,
and the `-p` value used in docker.
If you're using a [reverse proxy](../reverseProxy.md) such as Nginx for {{ project.name }} already, you could add this to your config's `server` block
```nginx
location /media/ {
# If you changed the port, be sure to change it here too
proxy_pass http://127.0.0.1:8000/;
}
```
Along with any additional config you already have, of course.
Alternative (and perhaps the better choice) would be to create a new domain, say `media.example.com` specifically for Imagor.
??? "Example config for `media.example.com` site"
```nginx
server {
# Change the server_name to reflect your true domain
server_name media.example.com;
add_header Last-Modified $date_gmt;
proxy_set_header Host $host;
proxy_pass_request_headers on;
proxy_set_header X-Forwarded-For $remote_addr;
location / {
# If you had changed the port, change it here as well
proxy_pass http://127.0.0.1:8000;
}
}
```
Our last step is to simply tell {{ project.name }} about Imagor. Just set the `cdn_imagorServerUrl` config value to your public endpoint for Imagor, wrapped in quotes.
For example, if you used the `/media` location in your existing nginx config, it will look something like `"https://your.{{ project.domain }}/media"`.
If you used a subdomain, it will look like `"https://media.your.{{ project.domain }}"`.
Don't include a trailing backslash.
Congrats! After a restart, you've now got Imagor resizing your images!

6
docs/setup/server/configuration/index.md
View File

@ -29,7 +29,7 @@ Arrays are represented by \_[number] in a config key. For example, multiple `gui
| cdn_endpointPublic | null | string | CDN endpoint. See gateway_endpointPublic |
| cdn_resizeHeightMax | 1000 | number | Maximum image resize height for embeds. |
| cdn_resizeWidthMax | 1000 | number | Maximum image resize width for embeds. |
| [cdn_imagorServerUrl](imagor.md) | null | string | Imagor instance endpoint for external image resizing. |
| [cdn_imagorServerUrl](imageProxy.md) | null | string | Endpoint to proxy embed images, e.g. using Imagor. |
| api_defaultVersion | 9 | string | API version to use when not specified |
| api_activeVersions_0 | 6, 7, 8, 9 | string[] | Allowed API version numbers. [Array](#array-types). |
| api_endpointPublic | null | string | Endpoint the API is available at, usually `<Base URL>/api` |
@ -89,8 +89,8 @@ Arrays are represented by \_[number] in a config key. For example, multiple `gui
| security_captcha_secret | null | string | Captcha service secret |
| security_twoFactor_generateBackupCodes | true | boolean | Whether to generate backup codes for MFA users |
| security_autoUpdate | true | boolean | Automatically updates NPM packages daily. Currently unused |
| security_requestSignature | Secret secret | string | The signature required for CDN or [Imagor](imagor.md) usage |
| security_jwtSecret | Secure secret | string | The secret used for user token generation |
| security_requestSignature | Random secret | string | The signature required for CDN or [Imagor](imageProxy.md) usage |
| security_jwtSecret | Random secret | string | The secret used for user token generation |
| [security_forwardedFor](../reverseProxy.md) | null | string | HTTP header for user's real IP |
| security_ipdataApiKey | {{ project.name }} IPdata key | string | API key used for IP geolocation and proxy detection |
| security_mfaBackupCodeCount | 10 | number | Number of MFA backup codes to generate |

4
docs/setup/server/database.md
View File

@ -6,10 +6,10 @@ You may delete this file to regenerate a new SQLite database on the next server
(or through `npm run sync:db`).
However, if you plan to run an instance with any sort of demand, you'd best set up a more Proper™ database
such as MariaDB or PostreSQL, which are both popular choices within the community.
such as MariaDB or PostgreSQL, which are both popular choices within the community.
We won't go into the setup of these servers here, given the scope of our documentation,
but to configure {{ project.name }} to use your shiny new database, simply set the `DATABASE` [environment variable](configuration/env.md)
to your new database connection string.
Usually, such a string will look something like `type://username:password@your-IP/databaseName`
Usually, such a string will look something like `type://username:password@host-IP:port/databaseName`

4
docs/setup/server/maintenance/restart.md
View File

@ -2,7 +2,7 @@
{{ project.name }}-server must restart in the following cases:
- The .env file or config (database or [CONFIG_PATH](/setup/server/configuration) has changed.
- The .env file or config (database or [CONFIG_PATH](/setup/server/configuration)) has changed.
- Files in the `client_test`, `preload-plugins` folders of inside `assets` have changed.
- Files inside `src` have changed (_remember to `npm run build`_).
- The `assets/schemas.json` file has changed.
@ -10,8 +10,6 @@
For all other cases, you do not need to restart the server. For example:
- Running `npm run generate:client`
- Running `npm run generate:changelog`
- Editing any table in the database other than `config`. For example, to edit a `users` [`rights`](/setup/server/configuration/rabbitmq)
- Database connection was lost (this should auto reconnect)

2
docs/setup/server/maintenance/updating.md
View File

@ -1,4 +1,4 @@
If you had followed the [setup guide](index.md), you'll have installed {{ project.name }} using [Git](https://git-scm.com/).
If you followed the [setup guide](index.md), you'll have installed {{ project.name }} using [Git](https://git-scm.com/).
Thus, you can update the server by running `git pull` in the `{{ project.name.lower() }}-server` directory.
If you had made any changes locally, you may run into merge conflicts,

20
docs/setup/server/migatingFromStaging.md
View File

@ -1,20 +0,0 @@
# Migrating from Staging
'Staging' refers to [this branch]({{ repositories.base_url }}/{{ repositories.server }}/tree/staging) of {{ project.name.lower() }}-server,
which is deprecated in favour of [the refactor branch]({{ repositories.base_url }}/{{ repositories.server }}/tree/maddy/refactor) (if this is a 404, its just become the main branch now).
!!! warning "If you were using SQLite, you will be unable to migrate to the new version. SQLite is primarily for testing and development purposes, and it should not be used for production."
To migrate from Staging:
1. Download the new version of {{ project.name.lower() }}-server
`git clone {{ repositories.base_url }}/{{ repositories.server }}.git`
2. Copy your `.env` file from your previous installation to the new `{{ project.name.lower() }}-server` folder.
3. Copy any user data, specifically the `files` directory, to the new project root.
4. `npm i` in new installation
5. `npm run setup`
6. `npm run migrate-from-staging` to run the migrations on your database
7. `npm run start` to start the server
Make sure you modify any existing scripts of yours to use the new path,
or just rename the old one and move the new one in its place.

4
docs/setup/server/npmScripts.md
View File

@ -15,6 +15,10 @@ Syncronise the database schema between {{ project.name }} source code and your d
Builds the {{ project.name }} source code
## `watch`
Starts the compilation of the {{ project.name }} source code in watch mode, automatically recompiling when changes are detected.
## `start`
Starts the bundled server. API, Gateway, and CDN run under the same process when using bundle, using the same port.

4
docs/setup/server/security/index.md
View File

@ -4,8 +4,8 @@ There are various security measures available to instance owners.
## [Rights](rights.md)
## [Captchas](captcha.md)
## [CAPTCHAs](captcha.md)
## [Registration Tokens](regTokens.md)
## [Rate Limits](limits.md)
## [Rate Limiting](limits.md)

1
docs/setup/server/security/regTokens.md
View File

@ -2,7 +2,6 @@
Registration tokens are a one-time use token for allowing a new user registration to bypass various restrictions:
- Bypass `register_allowNewRegistrations = false`
- Bypass `register_disabled = true`
- Bypass [captchas](captcha.md)
- Bypass `register_allowMultipleAccounts = false`