Everything on, phase 1: geolocation fetches itself, the deploy signs in as @thepra, the crawler is on, sign-up by invitation
Owner decisions (2026-10-04, recorded in docs/ROADMAP.md): production runs everything that is built, and nothing waits
on a person running a command.
- Geolocation updates itself. GeoUpdater, a hosted service, checks daily whether each DB-IP Lite database was built this
month. If not, it fetches this month's, or last month's early in the month. It installs a file only once it opens as
the right kind of database, then swaps it in atomically, and the locator reloads at once. Lookups now run under the
lock, so a reload can no longer dispose a reader mid-lookup. The systemd timer, its script and their setup.sh lines
are gone: the root step they needed never happened, and none is needed now. /stargazing names the database in use.
- The admin CLI runs after the app is built, with every service and nothing started.
- `create-root <login> [--admin]` takes the password on stdin; it is how the first login is made while sign-up is
closed.
- `smoke <persona>` keeps the root `deploy-smoke` and an undiscoverable persona, and gives the root a new password
on every run.
- The deploy signs in as @thepra. It runs the CLI, gets a token through the real OAuth flow (tools/smoke/oauth.sh,
moved out of the pasture's privapub_token, which now uses it), checks the signed-in API and that @thepra is
undiscoverable, then revokes the token. PRIVAPUB_SMOKE_TOKEN is gone.
- The deploy also fails when:
- NodeInfo and the instance API disagree about registrations;
- /stargazing does not say the crawler is on;
- the geolocation databases are missing or more than 40 days old.
- The crawler is on in production, seeded with ten large servers of different kinds. FEDERATION.md now describes it
and how to opt out.
- One registrations switch (Registrations:Mode, default Invitations; Open in tests and the pasture). It is read by
open sign-up (403 when closed), NodeInfo `openRegistrations`, and v1 and v2 of the instance API, so they can no longer
disagree. Before, NodeInfo said open and the instance API said closed. Group invitations always work, so
invites_enabled is true.
- A persona edit through /clientapi no longer resets what the Mastodon API set (discoverable, locked, quote policy…):
the theme is merged into the settings instead of replacing them.
650 tests pass. The deploy's smoke step was rehearsed against the pasture's PrivaPub.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
This commit is contained in:
1 parent
cd5eb25948
commit
5f56681c01
33 files changed
+907
-155
No files matched your search
@@ -90,8 +90,6 @@ jobs:
|
||||
fi
|
||||
|
||||
- name: Verify what is being served
|
||||
env:
|
||||
SMOKE_TOKEN: ${{ secrets.PRIVAPUB_SMOKE_TOKEN }}
|
||||
run: |
|
||||
served=$(curl -fsS "$PUBLIC_URL/build.json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('commit',''))")
|
||||
[ "$served" = "$BUILD_COMMIT" ] || { echo "::error::served build is '$served', expected '$BUILD_COMMIT'"; exit 1; }
|
||||
@@ -106,6 +104,39 @@ jobs:
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/activity+json' \
|
||||
--data '{"type":"Follow","actor":"https://example.org/users/x","object":"'"$PUBLIC_URL"'/peasants/privapub"}' "$PUBLIC_URL/human-centipede")
|
||||
[ "$code" = "401" ] || { echo "::error::an unsigned inbox POST answered $code"; exit 1; }
|
||||
tools/smoke/mastodon-api.sh "$PUBLIC_URL" "$SMOKE_TOKEN"
|
||||
[ -n "$SMOKE_TOKEN" ] || echo "::warning::PRIVAPUB_SMOKE_TOKEN is not set, so the signed-in API was not checked"
|
||||
tools/smoke/mastodon-api.sh "$PUBLIC_URL"
|
||||
open=$(curl -fsS "$PUBLIC_URL/nodeinfo/2.1" | python3 -c "import json,sys; print(json.load(sys.stdin)['openRegistrations'])")
|
||||
enabled=$(curl -fsS "$PUBLIC_URL/api/v2/instance" | python3 -c "import json,sys; print(json.load(sys.stdin)['registrations']['enabled'])")
|
||||
[ "$open" = "$enabled" ] || { echo "::error::NodeInfo says registrations are $open, the instance API $enabled"; exit 1; }
|
||||
curl -fsS "$PUBLIC_URL/stargazing" | grep -q 'The crawler is <strong>on</strong>' \
|
||||
|| { echo "::error::/stargazing does not say the crawler is on"; exit 1; }
|
||||
echo "::notice::serving $served"
|
||||
|
||||
# @thepra is the deploy's own persona (owner decision, 2026-10-04): the server's CLI makes or keeps it, undiscoverable,
|
||||
# and gives its root a new password on every deploy, so no secret is stored and nobody has to create anything.
|
||||
- name: Sign in as @thepra and check the signed-in API
|
||||
run: |
|
||||
creds=$(cd "$WEB_ROOT" && ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin smoke thepra | grep -E '^deploy-smoke [^ ]+$' | tail -1)
|
||||
[ -n "$creds" ] || { echo "::error::the smoke persona could not be prepared"; exit 1; }
|
||||
read -r login password <<< "$creds"
|
||||
echo "::add-mask::$password"
|
||||
read -r token cid cs <<< "$(tools/smoke/oauth.sh "$PUBLIC_URL" "$login" "$password" thepra read)"
|
||||
echo "::add-mask::$token"
|
||||
tools/smoke/mastodon-api.sh "$PUBLIC_URL" "$token"
|
||||
discoverable=$(curl -fsS -H 'Accept: application/activity+json' "$PUBLIC_URL/peasants/thepra" | python3 -c "import json,sys; print(json.load(sys.stdin).get('discoverable'))")
|
||||
[ "$discoverable" = "False" ] || { echo "::error::@thepra is discoverable"; exit 1; }
|
||||
curl -fsS -o /dev/null -X POST "$PUBLIC_URL/oauth/revoke" --data-urlencode "token=$token" \
|
||||
--data-urlencode "client_id=$cid" --data-urlencode "client_secret=$cs"
|
||||
echo "::notice::signed in as @thepra"
|
||||
|
||||
# The server fetches DB-IP Lite itself (GeoUpdater) about 30 s after it starts and then daily; the deploy only checks.
|
||||
- name: Geolocation databases are current
|
||||
run: |
|
||||
for kind in city asn; do
|
||||
db="/var/lib/privapub/geo/dbip-$kind-lite.mmdb"
|
||||
for i in $(seq 1 60); do [ -s "$db" ] && break; sleep 10; done
|
||||
[ -s "$db" ] || { echo "::error::$db is missing: the server has not fetched DB-IP Lite"; exit 1; }
|
||||
days=$(( ($(date +%s) - $(stat -c %Y "$db")) / 86400 ))
|
||||
[ "$days" -le 40 ] || { echo "::error::$db was installed $days days ago"; exit 1; }
|
||||
done
|
||||
curl -fsS "$PUBLIC_URL/stargazing" | grep -o 'DB-IP Lite [0-9-]*' | head -1 | sed 's/^/::notice::locating with /'
|
||||
@@ -136,18 +136,27 @@ curl-able. Outbound fetches only go to https DNS names resolving to public addre
|
||||
`Federation__AllowPrivateNetworks=true`, `Federation__AllowPlainHttp=true` and `Federation__AcceptAnyCertificate=true`,
|
||||
which startup refuses in Production.
|
||||
|
||||
Promoting an admin on the box (signing up as "admin" grants nothing):
|
||||
The admin CLI runs with every service built but nothing started (no Kestrel, no hosted services), so it can run next to
|
||||
the live service. Sign-up is closed in production (invitations only), so the first login is made here; promoting is how
|
||||
an admin is made (signing up as "admin" grants nothing):
|
||||
|
||||
```bash
|
||||
cd /var/www/privapub.thepra.dev && sudo -u www-data ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin promote <root>
|
||||
cd /var/www/privapub.thepra.dev
|
||||
echo '<password>' | ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin create-root <login> --admin # password on stdin
|
||||
ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin promote|demote <root>
|
||||
ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin smoke <persona> # the deploy's: prints "deploy-smoke <new password>"
|
||||
```
|
||||
|
||||
The deploy runs them as `build-runner`, which owns the published files, reads `appsettings.Production.json` through
|
||||
group www-data and reaches the private mongod; `sudo -u www-data` works too.
|
||||
|
||||
## Federation invariants
|
||||
|
||||
1. **Every outbound request goes through `IFederationHttp`.** Its handler resolves the name itself and connects only
|
||||
to public addresses; redirects are followed by hand (three at most, each re-checked); bodies are capped at 1 MB;
|
||||
only JSON media types are read; a refused URL is not asked again for five minutes. Never create another
|
||||
`HttpClient` for federation.
|
||||
`HttpClient` for federation. The only other outbound traffic is SMTP and `GeoUpdater`'s monthly DB-IP Lite download
|
||||
(its own `geo` client, a fixed HTTPS host, size-capped, the file checked before it is swapped in).
|
||||
2. **Every fetch is signed by the instance actor** (`privapub`), never by a persona; deliveries are signed by the acting
|
||||
avatar or group. Both are draft-cavage rsa-sha256 over `(request-target) host date` (+ `digest` on bodies).
|
||||
3. **A remote document is believed only from its own address.** `RemoteActorService.FetchObject` requires the
|
||||
@@ -446,10 +455,17 @@ the owner's GoToSocial account.**
|
||||
`deploy.yml`: tests, self-contained linux-x64 publish, snapshot and `mongodump` to `/var/backups/privapub.thepra.dev`,
|
||||
stop → rsync → start, a `127.0.0.1:6970/build.json` health loop with rollback, then public checks (actor, NodeInfo,
|
||||
Swagger 404, inbox junk 400, unsigned 401) and `tools/smoke/mastodon-api.sh` (app registration, client credentials,
|
||||
discovery, instance, public timeline; pass a persona token as a second argument to check the signed-in side).
|
||||
discovery, instance, public timeline). Then it checks that what production should be running is running:
|
||||
- NodeInfo and the instance API agree on registrations (closed, invitations only);
|
||||
- `/stargazing` says the crawler is on;
|
||||
- **@thepra signs in**: `PrivaPub admin smoke thepra` gives the root `deploy-smoke` a new password, `tools/smoke/oauth.sh`
|
||||
runs the real OAuth flow (the pasture's `privapub_token` uses the same script), the signed-in API is checked, the
|
||||
persona must be undiscoverable, and the token is revoked;
|
||||
- the DB-IP Lite databases, which the server fetches itself (`GeoUpdater`), exist and are at most 40 days old.
|
||||
- **The box:** Max (`nuvola.xyz`). Unit `privapub` runs as www-data from `/var/www/privapub.thepra.dev` with
|
||||
`ASPNETCORE_ENVIRONMENT=Production`.
|
||||
- **One-time root setup:** `deploy/max/setup.sh`, run through `../arasaka.software/tools/max/run.sh`.
|
||||
- **One-time root setup:** `deploy/max/setup.sh`, run through `../arasaka.software/tools/max/run.sh` (directories, the
|
||||
runner's sudoers line, the units, nginx and the certificate). Nothing that runs later needs it again.
|
||||
- **Config:** `appsettings.Production.json` is committed and deployed, **secrets included, by the owner's convention**
|
||||
(the same as arasaka.software). A hand edit on the box is lost at the next deploy.
|
||||
- **Secrets drift:** `AppConfigurationService` copies `AppConfiguration` into Mongo on first boot and reads the stored
|
||||
|
||||
@@ -162,6 +162,22 @@ The page's OpenGraph and Twitter tags give the title, description and image; the
|
||||
days and shared by every account on the server. Images are served to clients only through PrivaPub's media proxy.
|
||||
`Federation:FetchLinkPreviews=false` turns page fetching off.
|
||||
|
||||
## Server descriptions and the crawler
|
||||
|
||||
PrivaPub keeps statistics about servers, never about accounts (see `/stargazing` on the server).
|
||||
- **Describing a server:** a server it exchanges activities with is described at most once a week, from:
|
||||
- `/.well-known/nodeinfo` and the NodeInfo document it links;
|
||||
- `/api/v2/instance`, falling back to `/api/v1/instance`.
|
||||
|
||||
These requests are unsigned, because they are not ActivityPub documents.
|
||||
- **The crawler** is on at privapub.thepra.dev. It identifies as
|
||||
`PrivaPub-Stargazer/<version> (+https://privapub.thepra.dev/stargazing)`.
|
||||
- **What it reads:** `/robots.txt`, then the documents above and `/api/v1/instance/peers`, and nothing else: no
|
||||
accounts, posts or directories.
|
||||
- **How often:** one server a minute, each at most weekly.
|
||||
- **Opting out:** in robots.txt, the crawler obeys the group `PrivaPub-Stargazer`, then `PrivaPub`, then `*`. A
|
||||
robots.txt that answers with a server error or times out also keeps it out, as do this server's domain blocks.
|
||||
|
||||
## Local-only posts
|
||||
|
||||
Posts with a location (shown to nearby users of this server) never leave the server, in any form.
|
||||
|
||||
@@ -154,6 +154,34 @@ namespace PrivaPub.Tests.Http
|
||||
Assert.DoesNotContain(persona.Root.UserName, update.ToJsonString());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_theme_change_here_keeps_what_the_mastodon_api_set()
|
||||
{
|
||||
var persona = await _host.Persona(await _host.SignUp("keep"), "keep");
|
||||
await DB.Default.Update<Avatar>().MatchID(persona.Id)
|
||||
.Modify(a => a.Settings.IsDiscoverable, false)
|
||||
.Modify(a => a.Settings.IsLocked, true)
|
||||
.Modify(a => a.Settings.QuotePolicy, QuotePolicies.Nobody)
|
||||
.ExecuteAsync(TestContext.Current.CancellationToken);
|
||||
using var client = _host.As(persona.Root.Jwt);
|
||||
|
||||
var response = await client.PostJson("/clientapi/avatar/private/update", new
|
||||
{
|
||||
avatarId = persona.Id,
|
||||
name = "Kept",
|
||||
biography = "same settings",
|
||||
settings = new { isDefault = false, darkThemeIndexColour = 100, themeIsDarkMode = true }
|
||||
});
|
||||
|
||||
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
||||
var settings = (await DB.Default.Find<Avatar>().MatchID(persona.Id).ExecuteFirstAsync(TestContext.Current.CancellationToken)).Settings;
|
||||
Assert.True(settings.ThemeIsDarkMode);
|
||||
Assert.Equal(100, settings.DarkThemeIndexColour);
|
||||
Assert.False(settings.IsDiscoverable);
|
||||
Assert.True(settings.IsLocked);
|
||||
Assert.Equal(QuotePolicies.Nobody, settings.QuotePolicy);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Updating_another_roots_persona_is_refused()
|
||||
{
|
||||
|
||||
@@ -28,7 +28,8 @@ namespace PrivaPub.Tests.Http
|
||||
Assert.Equal("4.2.0 (compatible; PrivaPub)", v1.Body.Text("version"));
|
||||
Assert.Equal(InstanceController.Version, v1.Body.Text("version"));
|
||||
Assert.Equal(PrivaPubHost.Host, v1.Body.Text("uri"));
|
||||
Assert.False(v1.Body.Flag("registrations"));
|
||||
Assert.True(v1.Body.Flag("registrations"));
|
||||
Assert.True(v1.Body.Flag("invites_enabled"));
|
||||
Assert.True(v1.Body["stats"].Number("user_count") >= 0);
|
||||
Assert.Equal(5000, v1.Body["configuration"]!["statuses"].Number("max_characters"));
|
||||
Assert.Equal(4, v1.Body["configuration"]!["polls"].Number("max_options"));
|
||||
@@ -41,7 +42,8 @@ namespace PrivaPub.Tests.Http
|
||||
Assert.Equal(InstanceController.Version, v2.Body.Text("version"));
|
||||
Assert.Equal(PrivaPubHost.Host, v2.Body.Text("domain"));
|
||||
Assert.Equal($"wss://{PrivaPubHost.Host}", v2.Body["configuration"]!["urls"].Text("streaming"));
|
||||
Assert.False(v2.Body["registrations"].Flag("enabled"));
|
||||
Assert.True(v2.Body["registrations"].Flag("enabled"));
|
||||
Assert.True((await anonymous.Get("/nodeinfo/2.1")).Ok().Body.Flag("openRegistrations"));
|
||||
Assert.False(v2.Body["configuration"]!["translation"].Flag("enabled"));
|
||||
Assert.Equal(7, v2.Body["api_versions"].Number("mastodon"));
|
||||
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
using Microsoft.AspNetCore.Hosting;
|
||||
using Microsoft.AspNetCore.Mvc.Testing;
|
||||
|
||||
using PrivaPub.Infrastructure;
|
||||
using PrivaPub.Tests.Support;
|
||||
using PrivaPub.Tests.Support.Host;
|
||||
|
||||
using System.Net;
|
||||
|
||||
namespace PrivaPub.Tests.Http
|
||||
{
|
||||
[Trait("Category", "Integration")]
|
||||
public sealed class RegistrationsTests : IAsyncLifetime
|
||||
{
|
||||
WebApplicationFactory<Program> _closed;
|
||||
|
||||
public async ValueTask InitializeAsync()
|
||||
{
|
||||
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
|
||||
var shared = await PrivaPubHost.Shared();
|
||||
_closed = shared.WithWebHostBuilder(builder => builder.UseSetting("Registrations:Mode", nameof(RegistrationMode.Invitations)));
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (_closed != default)
|
||||
await _closed.DisposeAsync();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Invitations_only_refuses_sign_up_and_every_document_says_closed()
|
||||
{
|
||||
using var client = _closed.CreateClient(new WebApplicationFactoryClientOptions { BaseAddress = new Uri(PrivaPubHost.Base + "/"), AllowAutoRedirect = false });
|
||||
client.DefaultRequestHeaders.Add(PrivaPubHost.ClientHeader, $"10.9.{Random.Shared.Next(256)}.{Random.Shared.Next(1, 255)}");
|
||||
|
||||
var signUp = await client.PostJson("/clientapi/user/signup", new { userName = $"closed{Guid.NewGuid():N}"[..20], password = "Closed-Door-Pass-1" });
|
||||
var nodeInfo = (await client.Get("/nodeinfo/2.1")).Ok().Body;
|
||||
var v1 = (await client.Get("/api/v1/instance")).Ok().Body;
|
||||
var v2 = (await client.Get("/api/v2/instance")).Ok().Body;
|
||||
|
||||
Assert.Equal(HttpStatusCode.Forbidden, signUp.StatusCode);
|
||||
Assert.Contains("invitation", await signUp.Content.ReadAsStringAsync(TestContext.Current.CancellationToken));
|
||||
Assert.False(nodeInfo.Flag("openRegistrations"));
|
||||
Assert.False(v1.Flag("registrations"));
|
||||
Assert.True(v1.Flag("invites_enabled"));
|
||||
Assert.False(v2["registrations"].Flag("enabled"));
|
||||
Assert.Equal(RegistrationOptions.ClosedMessage, v2["registrations"].Text("message"));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,5 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.ClientModels;
|
||||
@@ -37,8 +39,8 @@ namespace PrivaPub.Tests.Infrastructure
|
||||
{
|
||||
var root = await _host.SignUp("promote");
|
||||
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "promote", root.UserName.ToUpperInvariant() }));
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "promote", root.UserName }));
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "promote", root.UserName.ToUpperInvariant() }, _host.Services));
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "promote", root.UserName }, _host.Services));
|
||||
|
||||
Assert.Equal(new[] { Policies.IsAdmin, Policies.IsModerator, Policies.IsUser }.Order(), await StoredPolicies(root));
|
||||
}
|
||||
@@ -47,10 +49,10 @@ namespace PrivaPub.Tests.Infrastructure
|
||||
public async Task Demote_leaves_a_plain_user()
|
||||
{
|
||||
var root = await _host.SignUp("demote");
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "promote", root.UserName }));
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "promote", root.UserName }, _host.Services));
|
||||
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "demote", root.UserName }));
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "demote", root.UserName }));
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "demote", root.UserName }, _host.Services));
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "demote", root.UserName }, _host.Services));
|
||||
|
||||
Assert.Equal(new[] { Policies.IsUser }, await StoredPolicies(root));
|
||||
}
|
||||
@@ -58,8 +60,8 @@ namespace PrivaPub.Tests.Infrastructure
|
||||
[Fact]
|
||||
public async Task An_unknown_root_exits_1()
|
||||
{
|
||||
Assert.Equal(1, await AdminCommands.Run(new[] { "promote", $"nobody{Guid.NewGuid():N}" }));
|
||||
Assert.Equal(1, await AdminCommands.Run(new[] { "demote", $"nobody{Guid.NewGuid():N}" }));
|
||||
Assert.Equal(1, await AdminCommands.Run(new[] { "promote", $"nobody{Guid.NewGuid():N}" }, _host.Services));
|
||||
Assert.Equal(1, await AdminCommands.Run(new[] { "demote", $"nobody{Guid.NewGuid():N}" }, _host.Services));
|
||||
}
|
||||
|
||||
public static TheoryData<string[]> Misuses() => new()
|
||||
@@ -75,7 +77,7 @@ namespace PrivaPub.Tests.Infrastructure
|
||||
[MemberData(nameof(Misuses))]
|
||||
public async Task Misuse_exits_2(string[] args)
|
||||
{
|
||||
Assert.Equal(2, await AdminCommands.Run(args));
|
||||
Assert.Equal(2, await AdminCommands.Run(args, _host.Services));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -84,7 +86,7 @@ namespace PrivaPub.Tests.Infrastructure
|
||||
var root = await _host.SignUp("policies");
|
||||
Assert.Equal(HttpStatusCode.Forbidden, await AdminRoute(root));
|
||||
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "promote", root.UserName }));
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "promote", root.UserName }, _host.Services));
|
||||
Assert.Equal(HttpStatusCode.OK, await AdminRoute(root));
|
||||
using (var client = _host.As(root.Jwt))
|
||||
{
|
||||
@@ -92,8 +94,54 @@ namespace PrivaPub.Tests.Infrastructure
|
||||
Assert.Contains(Policies.IsAdmin, refreshed["policies"]!.AsArray().Select(p => p!.GetValue<string>()));
|
||||
}
|
||||
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "demote", root.UserName }));
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "demote", root.UserName }, _host.Services));
|
||||
Assert.Equal(HttpStatusCode.Forbidden, await AdminRoute(root));
|
||||
}
|
||||
|
||||
async Task<HttpStatusCode> LogIn(string login, string password)
|
||||
{
|
||||
using var client = _host.Client();
|
||||
return (await client.PostJson("/clientapi/user/login", new { userName = login, password })).StatusCode;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Create_root_makes_a_login_from_a_password_on_standard_input()
|
||||
{
|
||||
using var scope = _host.Services.CreateScope();
|
||||
var login = $"made{Guid.NewGuid():N}"[..20];
|
||||
var output = new StringWriter();
|
||||
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "create-root", login, "--admin" }, scope.ServiceProvider, new StringReader("Typed-On-Stdin-1\n"), output));
|
||||
Assert.Equal(1, await AdminCommands.Run(new[] { "create-root", login }, scope.ServiceProvider, new StringReader("Typed-On-Stdin-2\n"), new StringWriter()));
|
||||
Assert.Equal(1, await AdminCommands.Run(new[] { "create-root", $"weak{Guid.NewGuid():N}"[..20] }, scope.ServiceProvider, new StringReader("short\n"), new StringWriter()));
|
||||
Assert.Equal(2, await AdminCommands.Run(new[] { "create-root", login, "--root" }, scope.ServiceProvider, new StringReader("Typed-On-Stdin-3\n"), new StringWriter()));
|
||||
|
||||
Assert.Equal(HttpStatusCode.OK, await LogIn(login, "Typed-On-Stdin-1"));
|
||||
var stored = await DB.Default.Find<RootUser>().Match(u => u.UserName == login).ExecuteFirstAsync(TestContext.Current.CancellationToken);
|
||||
Assert.Contains(Policies.IsAdmin, stored.Policies);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Smoke_keeps_one_undiscoverable_persona_and_a_new_password_each_run()
|
||||
{
|
||||
using var scope = _host.Services.CreateScope();
|
||||
var persona = $"smoke{Guid.NewGuid():N}"[..20];
|
||||
var first = new StringWriter();
|
||||
var second = new StringWriter();
|
||||
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "smoke", persona }, scope.ServiceProvider, output: first));
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "smoke", persona }, scope.ServiceProvider, output: second));
|
||||
|
||||
var (login, oldPassword) = (first.ToString().Trim().Split(' ')[0], first.ToString().Trim().Split(' ')[1]);
|
||||
var newPassword = second.ToString().Trim().Split(' ')[1];
|
||||
Assert.Equal(AdminCommands.SmokeLogin, login);
|
||||
Assert.NotEqual(oldPassword, newPassword);
|
||||
Assert.Equal(HttpStatusCode.OK, await LogIn(login, newPassword));
|
||||
Assert.NotEqual(HttpStatusCode.OK, await LogIn(login, oldPassword));
|
||||
var avatars = await DB.Default.Find<Avatar>().Match(a => a.UserName == persona).ExecuteAsync(TestContext.Current.CancellationToken);
|
||||
var avatar = Assert.Single(avatars);
|
||||
Assert.False(avatar.Settings.IsDiscoverable);
|
||||
Assert.False(avatar.Settings.IsIndexable);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
using MaxMind.Db;
|
||||
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
|
||||
using PrivaPub.Infrastructure.Geo;
|
||||
using PrivaPub.Infrastructure.Statistics;
|
||||
using PrivaPub.Tests.Support;
|
||||
|
||||
using System.IO.Compression;
|
||||
|
||||
namespace PrivaPub.Tests.Infrastructure
|
||||
{
|
||||
public sealed class GeoUpdaterTests : IAsyncLifetime
|
||||
{
|
||||
static readonly DateTime Now = new(2026, 10, 4, 12, 0, 0, DateTimeKind.Utc);
|
||||
static readonly DateTime LastMonth = new(2026, 9, 2, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
readonly string _directory = Path.Combine(Path.GetTempPath(), $"privapub-geo-{Guid.NewGuid():N}");
|
||||
Peer _peer;
|
||||
GeoUpdater _updater;
|
||||
DbIpLocator _locator;
|
||||
|
||||
public async ValueTask InitializeAsync()
|
||||
{
|
||||
_peer = await Peer.Start();
|
||||
var options = new StaticOptions<StatisticsOptions>(new StatisticsOptions
|
||||
{
|
||||
GeoDirectory = _directory,
|
||||
Geo = new GeoOptions { DownloadBase = $"{_peer.A}/free" }
|
||||
});
|
||||
_locator = new DbIpLocator(options, NullLogger<DbIpLocator>.Instance);
|
||||
_updater = new GeoUpdater(new Clients(), options, _locator, NullLogger<GeoUpdater>.Instance);
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
_locator?.Dispose();
|
||||
await _peer.DisposeAsync();
|
||||
if (Directory.Exists(_directory))
|
||||
Directory.Delete(_directory, recursive: true);
|
||||
}
|
||||
|
||||
sealed class Clients : IHttpClientFactory
|
||||
{
|
||||
public HttpClient CreateClient(string name) => new();
|
||||
}
|
||||
|
||||
static byte[] Gzip(byte[] bytes)
|
||||
{
|
||||
using var output = new MemoryStream();
|
||||
using (var gzip = new GZipStream(output, CompressionLevel.Fastest))
|
||||
gzip.Write(bytes);
|
||||
return output.ToArray();
|
||||
}
|
||||
|
||||
void Publish(string kind, DateTime month, byte[] database) =>
|
||||
_peer.ServeFile($"/free/dbip-{kind}-lite-{month:yyyy-MM}.mmdb.gz", Gzip(database), "application/gzip");
|
||||
|
||||
static byte[] City(DateTime built) => MiniMmdb.Build("DBIP-City-Lite", built);
|
||||
static byte[] Asn(DateTime built) => MiniMmdb.Build("DBIP-ASN-Lite (compat=GeoLite2-ASN)", built);
|
||||
|
||||
DateTime Built(string kind)
|
||||
{
|
||||
using var reader = new Reader(Path.Combine(_directory, GeoUpdater.FileOf(kind)));
|
||||
return reader.Metadata.BuildDate;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Missing_databases_are_fetched_checked_and_read_at_once()
|
||||
{
|
||||
Publish("city", Now, City(Now));
|
||||
Publish("asn", Now, Asn(Now));
|
||||
|
||||
Assert.Equal(2, await _updater.Update(Now, TestContext.Current.CancellationToken));
|
||||
|
||||
Assert.Equal(Now.Date, Built("city").Date);
|
||||
Assert.Equal(Now.Date, Built("asn").Date);
|
||||
Assert.Equal("DB-IP Lite 2026-10", _locator.Source);
|
||||
Assert.Empty(Directory.GetFiles(_directory).Where(f => f.EndsWith(".part") || f.EndsWith(".new")));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Early_in_the_month_last_months_database_is_used_and_not_fetched_again()
|
||||
{
|
||||
Publish("city", LastMonth, City(LastMonth));
|
||||
Publish("asn", LastMonth, Asn(LastMonth));
|
||||
|
||||
Assert.Equal(2, await _updater.Update(Now, TestContext.Current.CancellationToken));
|
||||
var asked = _peer.Requests.Count;
|
||||
Assert.Equal(0, await _updater.Update(Now, TestContext.Current.CancellationToken));
|
||||
|
||||
Assert.Equal(LastMonth.Date, Built("city").Date);
|
||||
Assert.DoesNotContain(_peer.Requests.Skip(asked), r => r.Path.Contains("2026-09"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_current_database_asks_nothing()
|
||||
{
|
||||
Publish("city", Now, City(Now));
|
||||
Publish("asn", Now, Asn(Now));
|
||||
await _updater.Update(Now, TestContext.Current.CancellationToken);
|
||||
var asked = _peer.Requests.Count;
|
||||
|
||||
Assert.Equal(0, await _updater.Update(Now.AddDays(1), TestContext.Current.CancellationToken));
|
||||
|
||||
Assert.Equal(asked, _peer.Requests.Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_broken_or_wrong_database_is_refused_and_the_old_one_kept()
|
||||
{
|
||||
Publish("city", LastMonth, City(LastMonth));
|
||||
Publish("asn", LastMonth, Asn(LastMonth));
|
||||
await _updater.Update(LastMonth.AddDays(1), TestContext.Current.CancellationToken);
|
||||
Publish("city", Now, "not a database at all"u8.ToArray());
|
||||
Publish("asn", Now, City(Now));
|
||||
|
||||
Assert.Equal(0, await _updater.Update(Now, TestContext.Current.CancellationToken));
|
||||
|
||||
Assert.Equal(LastMonth.Date, Built("city").Date);
|
||||
Assert.Equal(LastMonth.Date, Built("asn").Date);
|
||||
Assert.Empty(Directory.GetFiles(_directory).Where(f => f.EndsWith(".part") || f.EndsWith(".new")));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -39,7 +39,7 @@ namespace PrivaPub.Tests.Support.Host
|
||||
public static async Task<Root> Admin(this PrivaPubHost host)
|
||||
{
|
||||
var root = await host.SignUp("admin");
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "promote", root.UserName }));
|
||||
Assert.Equal(0, await AdminCommands.Run(new[] { "promote", root.UserName }, host.Services));
|
||||
return await host.LogIn(root);
|
||||
}
|
||||
|
||||
|
||||
@@ -73,6 +73,8 @@ namespace PrivaPub.Tests.Support.Host
|
||||
["Federation:AllowPrivateNetworks"] = "true",
|
||||
["Federation:AllowPlainHttp"] = "true",
|
||||
["Federation:FetchLinkPreviews"] = "false",
|
||||
["Registrations:Mode"] = "Open",
|
||||
["Statistics:Geo:AutoUpdate"] = "false",
|
||||
["Media:Root"] = _mediaRoot,
|
||||
["Logging:LogLevel:Default"] = "Warning",
|
||||
["Serilog:MinimumLevel:Default"] = Environment.GetEnvironmentVariable("PRIVAPUB_TEST_LOGS") == "1" ? "Information" : "Fatal"
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
using System.Text;
|
||||
|
||||
namespace PrivaPub.Tests.Support
|
||||
{
|
||||
// The smallest MaxMind DB the reader accepts (https://maxmind.github.io/MaxMind-DB/): an IPv4 search tree of one node
|
||||
// whose two records both mean "no data", an empty data section, and the metadata map, which is all a test of the updater
|
||||
// needs to tell a city database from an ASN one and to read its build date.
|
||||
public static class MiniMmdb
|
||||
{
|
||||
public static byte[] Build(string databaseType, DateTime built)
|
||||
{
|
||||
using var output = new MemoryStream();
|
||||
const uint nodeCount = 1;
|
||||
for (var record = 0; record < 2; record++)
|
||||
output.Write(new byte[] { 0, 0, (byte)nodeCount });//24-bit records equal to node_count: not found
|
||||
output.Write(new byte[16]);//data section separator; the data section itself is empty
|
||||
output.Write(new byte[] { 0xAB, 0xCD, 0xEF });
|
||||
output.Write(Encoding.ASCII.GetBytes("MaxMind.com"));
|
||||
Map(output,
|
||||
("binary_format_major_version", w => UInt16(w, 2)),
|
||||
("binary_format_minor_version", w => UInt16(w, 0)),
|
||||
("build_epoch", w => UInt64(w, (ulong)new DateTimeOffset(built).ToUnixTimeSeconds())),
|
||||
("database_type", w => Text(w, databaseType)),
|
||||
("description", w => Map(w, ("en", x => Text(x, "a test database")))),
|
||||
("ip_version", w => UInt16(w, 4)),
|
||||
("languages", w => Array(w, x => Text(x, "en"))),
|
||||
("node_count", w => UInt32(w, nodeCount)),
|
||||
("record_size", w => UInt16(w, 24)));
|
||||
return output.ToArray();
|
||||
}
|
||||
|
||||
// type in the top three bits (0 and an extra byte for the extended types), size below 29 in the low five, or 29 and the
|
||||
// rest in one more byte (names longer than 28 characters need it)
|
||||
static void Control(Stream output, int type, int size)
|
||||
{
|
||||
var low = size < 29 ? size : 29;
|
||||
output.WriteByte((byte)((type <= 7 ? type : 0) << 5 | low));
|
||||
if (type > 7)
|
||||
output.WriteByte((byte)(type - 7));
|
||||
if (size >= 29)
|
||||
output.WriteByte((byte)(size - 29));
|
||||
}
|
||||
|
||||
static void Text(Stream output, string value)
|
||||
{
|
||||
var bytes = Encoding.UTF8.GetBytes(value);
|
||||
Control(output, 2, bytes.Length);
|
||||
output.Write(bytes);
|
||||
}
|
||||
|
||||
static void Unsigned(Stream output, int type, ulong value)
|
||||
{
|
||||
var bytes = BitConverter.GetBytes(value).Reverse().SkipWhile(b => b == 0).ToArray();
|
||||
Control(output, type, bytes.Length);
|
||||
output.Write(bytes);
|
||||
}
|
||||
|
||||
static void UInt16(Stream output, ushort value) => Unsigned(output, 5, value);
|
||||
static void UInt32(Stream output, uint value) => Unsigned(output, 6, value);
|
||||
static void UInt64(Stream output, ulong value) => Unsigned(output, 9, value);
|
||||
|
||||
static void Map(Stream output, params (string Key, Action<Stream> Value)[] entries)
|
||||
{
|
||||
Control(output, 7, entries.Length);
|
||||
foreach (var (key, value) in entries)
|
||||
{
|
||||
Text(output, key);
|
||||
value(output);
|
||||
}
|
||||
}
|
||||
|
||||
static void Array(Stream output, params Action<Stream>[] items)
|
||||
{
|
||||
Control(output, 11, items.Length);
|
||||
foreach (var item in items)
|
||||
item(output);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,11 +1,13 @@
|
||||
using Microsoft.AspNetCore.Authorization;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Api.Mastodon.Infrastructure;
|
||||
using PrivaPub.Domain.Privacy;
|
||||
using PrivaPub.Federation.Actors;
|
||||
using PrivaPub.Infrastructure;
|
||||
using PrivaPub.Models.User;
|
||||
|
||||
using PostEntity = PrivaPub.Models.Post.Post;
|
||||
@@ -18,12 +20,16 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
const string Description = "A small ActivityPub server where one private login keeps several unlinkable public personas.";
|
||||
|
||||
readonly ILocalActorService _localActors;
|
||||
readonly IOptionsMonitor<RegistrationOptions> _registrations;
|
||||
|
||||
public InstanceController(ILocalActorService localActors)
|
||||
public InstanceController(ILocalActorService localActors, IOptionsMonitor<RegistrationOptions> registrations)
|
||||
{
|
||||
_localActors = localActors;
|
||||
_registrations = registrations;
|
||||
}
|
||||
|
||||
bool Open => _registrations.CurrentValue.IsOpen;
|
||||
|
||||
string Domain => new Uri(_localActors.BaseAddress).Authority;
|
||||
|
||||
static object Statuses => new { max_characters = 5000, max_media_attachments = 4, characters_reserved_per_url = 23 };
|
||||
@@ -58,9 +64,9 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
stats = new { user_count = users, status_count = statuses, domain_count = domains },
|
||||
thumbnail = $"{_localActors.BaseAddress}/media/missing-header.png",
|
||||
languages = new[] { "en" },
|
||||
registrations = false,
|
||||
registrations = Open,
|
||||
approval_required = false,
|
||||
invites_enabled = false,
|
||||
invites_enabled = true,//a group invitation always creates an account
|
||||
configuration = new { accounts = new { max_featured_tags = 0 }, statuses = Statuses, media_attachments = MediaAttachments, polls = Polls },
|
||||
contact_account = default(object),
|
||||
rules = Array.Empty<object>()
|
||||
@@ -88,7 +94,7 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
polls = Polls,
|
||||
translation = new { enabled = false }
|
||||
},
|
||||
registrations = new { enabled = false, approval_required = false, message = default(string) },
|
||||
registrations = new { enabled = Open, approval_required = false, message = Open ? default : RegistrationOptions.ClosedMessage },
|
||||
contact = new { email = string.Empty, account = default(object) },
|
||||
rules = Array.Empty<object>()
|
||||
});
|
||||
|
||||
@@ -34,6 +34,7 @@ namespace PrivaPub.Controllers.ClientToServer
|
||||
readonly ILocalActorService LocalActors;
|
||||
readonly AuthTokenManager AuthTokenManager;
|
||||
readonly IOptionsMonitor<AppConfiguration> AppConfiguration;
|
||||
readonly IOptionsMonitor<RegistrationOptions> Registrations;
|
||||
readonly ILogger<RootUserController> Logger;
|
||||
readonly IStringLocalizer Localizer;
|
||||
|
||||
@@ -43,9 +44,11 @@ namespace PrivaPub.Controllers.ClientToServer
|
||||
ILocalActorService localActors,
|
||||
AuthTokenManager authTokenManager,
|
||||
IOptionsMonitor<AppConfiguration> appConfiguration,
|
||||
IOptionsMonitor<RegistrationOptions> registrations,
|
||||
IStringLocalizer<GenericRes> localizer,
|
||||
ILogger<RootUserController> logger)
|
||||
{
|
||||
Registrations = registrations;
|
||||
UsersService = usersService;
|
||||
GroupUsersService = groupUsersService;
|
||||
AvatarUsersService = avatarUsersService;
|
||||
@@ -63,6 +66,8 @@ namespace PrivaPub.Controllers.ClientToServer
|
||||
{
|
||||
if (User.Identity?.IsAuthenticated ?? false) return Redirect("/");
|
||||
var result = new WebResult();
|
||||
if (!Registrations.CurrentValue.IsOpen)
|
||||
return StatusCode(StatusCodes.Status403Forbidden, result.Invalidate(Localizer[RegistrationOptions.ClosedMessage], StatusCodes.Status403Forbidden));
|
||||
if (!ModelState.IsValid)
|
||||
return BadRequest(result.Invalidate(Localizer["Invalid model."]));
|
||||
try
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
using MongoDB.Entities;
|
||||
|
||||
@@ -12,6 +13,7 @@ using System.Text.Json.Nodes;
|
||||
using PostEntity = PrivaPub.Models.Post.Post;
|
||||
using PrivaPub.Domain.Privacy;
|
||||
using PrivaPub.Federation.Actors;
|
||||
using PrivaPub.Infrastructure;
|
||||
|
||||
namespace PrivaPub.Federation.Controllers
|
||||
{
|
||||
@@ -20,9 +22,11 @@ namespace PrivaPub.Federation.Controllers
|
||||
{
|
||||
readonly ILocalActorService _localActors;
|
||||
readonly DbEntities _dbEntities;
|
||||
readonly IOptionsMonitor<RegistrationOptions> _registrations;
|
||||
|
||||
public WellKnownController(ILocalActorService localActors, DbEntities dbEntities)
|
||||
public WellKnownController(ILocalActorService localActors, DbEntities dbEntities, IOptionsMonitor<RegistrationOptions> registrations)
|
||||
{
|
||||
_registrations = registrations;
|
||||
_localActors = localActors;
|
||||
_dbEntities = dbEntities;
|
||||
}
|
||||
@@ -102,7 +106,7 @@ namespace PrivaPub.Federation.Controllers
|
||||
["software"] = software,
|
||||
["protocols"] = new JsonArray("activitypub"),
|
||||
["services"] = new JsonObject { ["inbound"] = new JsonArray(), ["outbound"] = new JsonArray() },
|
||||
["openRegistrations"] = true,
|
||||
["openRegistrations"] = _registrations.CurrentValue.IsOpen,
|
||||
["usage"] = new JsonObject
|
||||
{
|
||||
["users"] = new JsonObject { ["total"] = users },
|
||||
|
||||
@@ -1,22 +1,49 @@
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.ClientModels;
|
||||
using PrivaPub.ClientModels.User;
|
||||
using PrivaPub.ClientModels.User.Avatar;
|
||||
using PrivaPub.Models.User;
|
||||
using PrivaPub.Services;
|
||||
using PrivaPub.Services.ClientToServer.Private;
|
||||
using PrivaPub.StaticServices;
|
||||
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
using System.Security.Cryptography;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Cli
|
||||
{
|
||||
// `PrivaPub admin ...` runs with the whole server built but not started: no Kestrel, no hosted services.
|
||||
public static class AdminCommands
|
||||
{
|
||||
const string Usage = "usage: PrivaPub admin promote|demote <root username>";
|
||||
public const string SmokeLogin = "deploy-smoke";
|
||||
|
||||
public static async Task<int> Run(string[] args)
|
||||
const string Usage = """
|
||||
usage: PrivaPub admin promote|demote <root>
|
||||
PrivaPub admin create-root <login> [--admin] the password is read from standard input
|
||||
PrivaPub admin smoke <persona> prints "<login> <password>" for the deploy's signed-in check
|
||||
""";
|
||||
|
||||
public static async Task<int> Run(string[] args, IServiceProvider services, TextReader input = default, TextWriter output = default)
|
||||
{
|
||||
if (args is not [("promote" or "demote") and var verb, var userName])
|
||||
input ??= Console.In;
|
||||
output ??= Console.Out;
|
||||
switch (args)
|
||||
{
|
||||
case [("promote" or "demote") and var verb, var userName]:
|
||||
return await Promote(verb == "promote", userName, output);
|
||||
case ["create-root", var login, .. var flags] when flags.All(f => f == "--admin"):
|
||||
return await CreateRoot(services, login, input.ReadLine(), flags.Contains("--admin"), output);
|
||||
case ["smoke", var persona]:
|
||||
return await Smoke(services, persona, output);
|
||||
default:
|
||||
Console.Error.WriteLine(Usage);
|
||||
return 2;
|
||||
}
|
||||
}
|
||||
|
||||
static async Task<int> Promote(bool promote, string userName, TextWriter output)
|
||||
{
|
||||
userName = userName.ToLowerInvariant();
|
||||
var user = await DB.Default.Find<RootUser>().Match(u => u.UserName == userName).ExecuteFirstAsync();
|
||||
if (user == default)
|
||||
@@ -26,14 +53,94 @@ namespace PrivaPub.Infrastructure.Cli
|
||||
}
|
||||
|
||||
user.Policies.RemoveAll(p => p is Policies.IsAdmin or Policies.IsModerator);
|
||||
if (verb == "promote")
|
||||
if (promote)
|
||||
user.Policies.AddRange(new[] { Policies.IsAdmin, Policies.IsModerator });
|
||||
if (!user.Policies.Contains(Policies.IsUser))
|
||||
user.Policies.Add(Policies.IsUser);
|
||||
user.UpdatedAt = DateTime.UtcNow;
|
||||
await DB.Default.SaveAsync(user);
|
||||
|
||||
Console.WriteLine($"{userName}: {string.Join(", ", user.Policies)}");
|
||||
output.WriteLine($"{userName}: {string.Join(", ", user.Policies)}");
|
||||
return 0;
|
||||
}
|
||||
|
||||
// With sign-up closed this is how the first root is made; group invitations make the rest.
|
||||
static async Task<int> CreateRoot(IServiceProvider services, string login, string password, bool admin, TextWriter output)
|
||||
{
|
||||
var form = new LoginForm { UserName = login, Password = password?.Trim() };
|
||||
var problems = new List<ValidationResult>();
|
||||
if (!Validator.TryValidateObject(form, new ValidationContext(form), problems, validateAllProperties: true))
|
||||
{
|
||||
Console.Error.WriteLine(string.Join(Environment.NewLine, problems.Select(p => p.ErrorMessage)));
|
||||
return 1;
|
||||
}
|
||||
var created = await services.GetRequiredService<IRootUsersService>().SignUpAsync(form);
|
||||
if (!created.IsValid)
|
||||
{
|
||||
Console.Error.WriteLine(created.ErrorMessage);
|
||||
return 1;
|
||||
}
|
||||
return admin ? await Promote(true, login, output) : await Report(login, output);
|
||||
}
|
||||
|
||||
static Task<int> Report(string login, TextWriter output)
|
||||
{
|
||||
output.WriteLine($"{login.ToLowerInvariant()}: created");
|
||||
return Task.FromResult(0);
|
||||
}
|
||||
|
||||
// The deploy's signed-in smoke check: a root nobody signs in to by hand, owning one undiscoverable persona. Every run
|
||||
// sets a new password and prints it, so no secret is kept anywhere and nothing waits on a person.
|
||||
static async Task<int> Smoke(IServiceProvider services, string personaName, TextWriter output)
|
||||
{
|
||||
personaName = personaName.ToLowerInvariant();
|
||||
var password = "Smoke-x" + Convert.ToHexStringLower(RandomNumberGenerator.GetBytes(24));
|
||||
var root = await DB.Default.Find<RootUser>().Match(u => u.UserName == SmokeLogin).ExecuteFirstAsync();
|
||||
if (root == default)
|
||||
{
|
||||
var created = await services.GetRequiredService<IRootUsersService>().SignUpAsync(new LoginForm { UserName = SmokeLogin, Password = password });
|
||||
if (!created.IsValid)
|
||||
{
|
||||
Console.Error.WriteLine(created.ErrorMessage);
|
||||
return 1;
|
||||
}
|
||||
root = await DB.Default.Find<RootUser>().Match(u => u.UserName == SmokeLogin).ExecuteFirstAsync();
|
||||
}
|
||||
else if (root.DeletedAt.HasValue || root.IsBanned)
|
||||
{
|
||||
Console.Error.WriteLine($"the root '{SmokeLogin}' is deleted or banned");
|
||||
return 1;
|
||||
}
|
||||
else
|
||||
await DB.Default.Update<RootUser>().MatchID(root.ID)
|
||||
.Modify(u => u.HashedPassword, services.GetRequiredService<IPasswordHasher>().Hash(password))
|
||||
.Modify(u => u.UpdatedAt, DateTime.UtcNow)
|
||||
.ExecuteAsync();
|
||||
|
||||
var owned = (await DB.Default.Find<RootToAvatar>().Match(ra => ra.RootId == root.ID).ExecuteAsync()).Select(ra => ra.AvatarId).ToList();
|
||||
var persona = await DB.Default.Find<Avatar>().Match(a => owned.Contains(a.ID) && a.UserName == personaName && !a.DeletionAt.HasValue).ExecuteFirstAsync();
|
||||
if (persona == default)
|
||||
{
|
||||
var inserted = await services.GetRequiredService<IPrivateAvatarUsersService>().InsertAvatar(new InsertAvatarForm
|
||||
{
|
||||
RootId = root.ID,
|
||||
UserName = personaName,
|
||||
Name = personaName,
|
||||
Biography = "The deploy signs in here to check that the Mastodon API works for a signed-in persona."
|
||||
});
|
||||
if (!inserted.IsValid)
|
||||
{
|
||||
Console.Error.WriteLine(inserted.ErrorMessage);
|
||||
return 1;
|
||||
}
|
||||
persona = await DB.Default.Find<Avatar>().MatchID(((ViewAvatar)inserted.Data).Id).ExecuteFirstAsync();
|
||||
}
|
||||
await DB.Default.Update<Avatar>().MatchID(persona.ID)
|
||||
.Modify(a => a.Settings.IsDiscoverable, false)
|
||||
.Modify(a => a.Settings.IsIndexable, false)
|
||||
.ExecuteAsync();
|
||||
|
||||
output.WriteLine($"{SmokeLogin} {password}");
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -25,8 +25,8 @@ namespace PrivaPub.Infrastructure.Geo
|
||||
|
||||
public sealed class DbIpLocator : IGeoLocator, IDisposable
|
||||
{
|
||||
public const string CityFile = "dbip-city-lite.mmdb";
|
||||
public const string AsnFile = "dbip-asn-lite.mmdb";
|
||||
public static readonly string CityFile = GeoUpdater.FileOf("city");
|
||||
public static readonly string AsnFile = GeoUpdater.FileOf("asn");
|
||||
static readonly TimeSpan CheckInterval = TimeSpan.FromMinutes(10);
|
||||
|
||||
readonly IOptionsMonitor<StatisticsOptions> _options;
|
||||
@@ -60,18 +60,17 @@ namespace PrivaPub.Infrastructure.Geo
|
||||
if (address == default || !IpRangeGuard.IsPublic(address))
|
||||
return default;
|
||||
Refresh();
|
||||
Reader city, asn;
|
||||
lock (_lock)
|
||||
{
|
||||
city = _city;
|
||||
asn = _asn;
|
||||
}
|
||||
if (city == default && asn == default)
|
||||
return default;
|
||||
Dictionary<string, object> place, network;
|
||||
try
|
||||
{
|
||||
var place = city?.Find<Dictionary<string, object>>(address);
|
||||
var network = asn?.Find<Dictionary<string, object>>(address);
|
||||
// under the lock, so a reload cannot dispose a reader in the middle of a lookup; lookups are rare
|
||||
lock (_lock)
|
||||
{
|
||||
if (_city == default && _asn == default)
|
||||
return default;
|
||||
place = _city?.Find<Dictionary<string, object>>(address);
|
||||
network = _asn?.Find<Dictionary<string, object>>(address);
|
||||
}
|
||||
var location = Section(place, "location");
|
||||
return new GeoFix(
|
||||
Text(Section(place, "country"), "iso_code"),
|
||||
@@ -88,6 +87,14 @@ namespace PrivaPub.Infrastructure.Geo
|
||||
}
|
||||
}
|
||||
|
||||
// Looks at the files again now instead of within the next ten minutes: GeoUpdater has just replaced one.
|
||||
public void Reload()
|
||||
{
|
||||
lock (_lock)
|
||||
_checkedAt = DateTime.MinValue;
|
||||
Refresh();
|
||||
}
|
||||
|
||||
void Refresh()
|
||||
{
|
||||
if (DateTime.UtcNow - _checkedAt < CheckInterval)
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
using MaxMind.Db;
|
||||
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
using PrivaPub.Infrastructure.Statistics;
|
||||
|
||||
using System.IO.Compression;
|
||||
using System.Net;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Geo
|
||||
{
|
||||
// Keeps the DB-IP Lite databases current by itself, so a deploy needs no timer and no root: once a day it checks
|
||||
// whether each database was built this month and, if not, fetches this month's (or, early in the month, last month's),
|
||||
// checks it opens as the right kind of database and swaps it in. This and SMTP are the server's only traffic that is not
|
||||
// federation, which is why it has its own client instead of IFederationHttp.
|
||||
public sealed class GeoUpdater : BackgroundService
|
||||
{
|
||||
public const string ClientName = "geo";
|
||||
static readonly string[] Kinds = { "city", "asn" };
|
||||
static readonly TimeSpan FirstWait = TimeSpan.FromSeconds(30);
|
||||
static readonly TimeSpan Interval = TimeSpan.FromHours(24);
|
||||
|
||||
readonly IHttpClientFactory _http;
|
||||
readonly IOptionsMonitor<StatisticsOptions> _options;
|
||||
readonly IGeoLocator _locator;
|
||||
readonly ILogger<GeoUpdater> _logger;
|
||||
|
||||
public GeoUpdater(IHttpClientFactory http, IOptionsMonitor<StatisticsOptions> options, IGeoLocator locator, ILogger<GeoUpdater> logger)
|
||||
{
|
||||
_http = http;
|
||||
_options = options;
|
||||
_locator = locator;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
public static string FileOf(string kind) => $"dbip-{kind}-lite.mmdb";
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(FirstWait, stoppingToken);
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
if (_options.CurrentValue.Geo.AutoUpdate)
|
||||
await Update(DateTime.UtcNow, stoppingToken);
|
||||
await Task.Delay(Interval + TimeSpan.FromMinutes(Random.Shared.Next(60)), stoppingToken);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
}
|
||||
}
|
||||
|
||||
// The number of databases replaced. A failure keeps the database already there and is tried again the next day.
|
||||
public async Task<int> Update(DateTime now, CancellationToken token)
|
||||
{
|
||||
var options = _options.CurrentValue;
|
||||
var replaced = 0;
|
||||
try
|
||||
{
|
||||
Directory.CreateDirectory(options.GeoDirectory);
|
||||
}
|
||||
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
_logger.LogWarning(ex, "Cannot create {Directory}; servers are not located", options.GeoDirectory);
|
||||
return 0;
|
||||
}
|
||||
foreach (var kind in Kinds)
|
||||
{
|
||||
var path = Path.Combine(options.GeoDirectory, FileOf(kind));
|
||||
var built = BuiltMonth(path);
|
||||
foreach (var month in new[] { Month(now), Month(now).AddMonths(-1) })
|
||||
{
|
||||
if (built >= month)
|
||||
break;
|
||||
try
|
||||
{
|
||||
if (await Install(kind, month, path, options, token))
|
||||
{
|
||||
replaced++;
|
||||
_logger.LogInformation("Installed DB-IP Lite {Kind} {Month:yyyy-MM}", kind, month);
|
||||
break;
|
||||
}
|
||||
}
|
||||
catch (Exception ex) when (ex is HttpRequestException or IOException or InvalidDatabaseException or InvalidDataException
|
||||
or TaskCanceledException && !token.IsCancellationRequested)
|
||||
{
|
||||
_logger.LogWarning(ex, "Could not install DB-IP Lite {Kind} {Month:yyyy-MM}", kind, month);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (replaced > 0 && _locator is DbIpLocator locator)
|
||||
locator.Reload();
|
||||
return replaced;
|
||||
}
|
||||
|
||||
static DateTime Month(DateTime at) => new(at.Year, at.Month, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
static DateTime BuiltMonth(string path)
|
||||
{
|
||||
if (!File.Exists(path))
|
||||
return DateTime.MinValue;
|
||||
try
|
||||
{
|
||||
using var reader = new Reader(path, FileAccessMode.MemoryMapped);
|
||||
return Month(reader.Metadata.BuildDate);
|
||||
}
|
||||
catch (Exception ex) when (ex is InvalidDatabaseException or IOException)
|
||||
{
|
||||
return DateTime.MinValue;
|
||||
}
|
||||
}
|
||||
|
||||
// False when that month is not published (yet); throws when it is published but cannot be used.
|
||||
async Task<bool> Install(string kind, DateTime month, string path, StatisticsOptions options, CancellationToken token)
|
||||
{
|
||||
var url = $"{options.Geo.DownloadBase.TrimEnd('/')}/dbip-{kind}-lite-{month:yyyy-MM}.mmdb.gz";
|
||||
var packed = path + ".gz.part";
|
||||
var unpacked = path + ".new";
|
||||
try
|
||||
{
|
||||
using (var response = await _http.CreateClient(ClientName).GetAsync(url, HttpCompletionOption.ResponseHeadersRead, token))
|
||||
{
|
||||
if (response.StatusCode == HttpStatusCode.NotFound)
|
||||
return false;
|
||||
response.EnsureSuccessStatusCode();
|
||||
await using var source = await response.Content.ReadAsStreamAsync(token);
|
||||
await using var target = File.Create(packed);
|
||||
await CopyCapped(source, target, options.Geo.MaxDownloadBytes, token);
|
||||
}
|
||||
await using (var source = new GZipStream(File.OpenRead(packed), CompressionMode.Decompress))
|
||||
await using (var target = File.Create(unpacked))
|
||||
await CopyCapped(source, target, options.Geo.MaxDatabaseBytes, token);
|
||||
using (var reader = new Reader(unpacked, FileAccessMode.MemoryMapped))
|
||||
if (!reader.Metadata.DatabaseType.Contains(kind, StringComparison.OrdinalIgnoreCase))
|
||||
throw new InvalidDatabaseException($"{url} is a {reader.Metadata.DatabaseType} database, not {kind}");
|
||||
File.Move(unpacked, path, overwrite: true);
|
||||
return true;
|
||||
}
|
||||
finally
|
||||
{
|
||||
File.Delete(packed);
|
||||
File.Delete(unpacked);
|
||||
}
|
||||
}
|
||||
|
||||
static async Task CopyCapped(Stream source, Stream target, long limit, CancellationToken token)
|
||||
{
|
||||
var buffer = new byte[81920];
|
||||
long total = 0;
|
||||
int read;
|
||||
while ((read = await source.ReadAsync(buffer, token)) > 0)
|
||||
{
|
||||
total += read;
|
||||
if (total > limit)
|
||||
throw new InvalidDataException($"more than {limit} bytes");
|
||||
await target.WriteAsync(buffer.AsMemory(0, read), token);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
namespace PrivaPub.Infrastructure
|
||||
{
|
||||
public enum RegistrationMode
|
||||
{
|
||||
Invitations,
|
||||
Open
|
||||
}
|
||||
|
||||
// Who may make a root login. NodeInfo, both instance API versions and the sign-up endpoint read this one switch, so they
|
||||
// cannot disagree. A group invitation always creates an account, whatever the mode.
|
||||
public class RegistrationOptions
|
||||
{
|
||||
public RegistrationMode Mode { get; set; } = RegistrationMode.Invitations;//owner decision (2026-10-04): invitations only
|
||||
|
||||
public bool IsOpen => Mode == RegistrationMode.Open;
|
||||
|
||||
public const string ClosedMessage = "Sign-up is by invitation only: ask someone here for a group invitation.";
|
||||
}
|
||||
}
|
||||
@@ -6,9 +6,19 @@ namespace PrivaPub.Infrastructure.Statistics
|
||||
public string GeoDirectory { get; set; } = "/var/lib/privapub/geo";
|
||||
public int PublicCityMinUsers { get; set; } = 10;
|
||||
public CrawlerOptions Crawler { get; set; } = new();
|
||||
public GeoOptions Geo { get; set; } = new();
|
||||
}
|
||||
|
||||
//owner decision: off by default; when on, one server a minute, each at most weekly, at most MaxHosts servers
|
||||
//owner decision (2026-10-04): the server keeps its DB-IP Lite databases current by itself
|
||||
public class GeoOptions
|
||||
{
|
||||
public bool AutoUpdate { get; set; } = true;
|
||||
public string DownloadBase { get; set; } = "https://download.db-ip.com/free";
|
||||
public long MaxDownloadBytes { get; set; } = 300L * 1024 * 1024;
|
||||
public long MaxDatabaseBytes { get; set; } = 1024L * 1024 * 1024;
|
||||
}
|
||||
|
||||
//owner decision: off by default (on in production since 2026-10-04); when on, one server a minute, each at most weekly, at most MaxHosts servers
|
||||
public class CrawlerOptions
|
||||
{
|
||||
public bool Enabled { get; set; }
|
||||
|
||||
@@ -41,7 +41,8 @@ namespace PrivaPub.Middleware
|
||||
{
|
||||
return service
|
||||
.Configure<MongoSettings>(configuration.GetSection(nameof(MongoSettings)))
|
||||
.Configure<AppConfiguration>(configuration.GetSection(nameof(AppConfiguration)));
|
||||
.Configure<AppConfiguration>(configuration.GetSection(nameof(AppConfiguration)))
|
||||
.Configure<PrivaPub.Infrastructure.RegistrationOptions>(configuration.GetSection("Registrations"));
|
||||
}
|
||||
public static IServiceCollection PrivaPubWorkersConfiguration(this IServiceCollection service)
|
||||
{
|
||||
@@ -107,8 +108,16 @@ namespace PrivaPub.Middleware
|
||||
.AddSingleton<IJobHandler, DeliveryJobHandler>()
|
||||
.AddHostedService<JobWorker>();
|
||||
}
|
||||
public static IServiceCollection PrivaPubStatisticsConfiguration(this IServiceCollection service, IConfiguration configuration) =>
|
||||
service
|
||||
public static IServiceCollection PrivaPubStatisticsConfiguration(this IServiceCollection service, IConfiguration configuration)
|
||||
{
|
||||
// not federation: the DB-IP download only (GeoUpdater), plain HTTPS to a fixed host
|
||||
service.AddHttpClient(GeoUpdater.ClientName, (provider, client) =>
|
||||
{
|
||||
var baseAddress = provider.GetRequiredService<IOptionsMonitor<AppConfiguration>>().CurrentValue.BackendBaseAddress?.TrimEnd('/');
|
||||
client.Timeout = TimeSpan.FromMinutes(15);
|
||||
client.DefaultRequestHeaders.UserAgent.ParseAdd($"PrivaPub/{BuildInfo.Ref} (+{baseAddress}/)");
|
||||
});
|
||||
return service
|
||||
.Configure<StatisticsOptions>(configuration.GetSection("Statistics"))
|
||||
.AddSingleton<InteractionSalts>()
|
||||
.AddSingleton<InteractionLedger>()
|
||||
@@ -116,10 +125,12 @@ namespace PrivaPub.Middleware
|
||||
.AddHostedService(services => services.GetRequiredService<InteractionLedger>())
|
||||
.AddSingleton<IJobHandler, RollupJob>()
|
||||
.AddSingleton<IGeoLocator, DbIpLocator>()
|
||||
.AddHostedService<GeoUpdater>()
|
||||
.AddSingleton<Domain.Statistics.StatisticsQueries>()
|
||||
.AddSingleton<IJobHandler, Federation.Crawler.CrawlPlanner>()
|
||||
.AddSingleton<IJobHandler, Federation.Crawler.InstanceCrawler>()
|
||||
.AddHostedService<StatisticsSchedule>();
|
||||
}
|
||||
|
||||
public static IServiceCollection PrivaPubAuthServicesConfiguration(this IServiceCollection service, IConfiguration configuration)
|
||||
{
|
||||
|
||||
+9
-6
@@ -87,12 +87,6 @@ try
|
||||
EntityMaps.Warm();
|
||||
await DB.Default.MigrateAsync<Program>();
|
||||
await Indexes.Create();
|
||||
|
||||
if (args is ["admin", ..])
|
||||
{
|
||||
Environment.ExitCode = await AdminCommands.Run(args[1..]);
|
||||
return;
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -111,6 +105,15 @@ try
|
||||
throw;
|
||||
}
|
||||
|
||||
// Commands get every service but start nothing: no Kestrel, no hosted services, no media directory. The deploy runs
|
||||
// them as its own user, which can read the configuration and reach the private mongod but owns no www-data directory.
|
||||
if (args is ["admin", ..])
|
||||
{
|
||||
using var scope = app.Services.CreateScope();
|
||||
Environment.ExitCode = await AdminCommands.Run(args[1..], scope.ServiceProvider);
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var localizationService = app.Services.GetService<RequestLocalizationOptionsService>();
|
||||
|
||||
@@ -73,7 +73,7 @@ namespace PrivaPub.Services.ClientToServer.Private
|
||||
if (!await _localActors.TryReserveUserName(userName, LocalActorKind.Person, newAvatar.ID, default))
|
||||
return result.Invalidate(_localizer["The username '{0}' is already take.", userName]);
|
||||
if (form.Settings is { IsDefault: false })
|
||||
newAvatar.Settings = ToSettings(form.Settings);
|
||||
Apply(newAvatar.Settings, form.Settings);
|
||||
|
||||
var actor = _localActors.FromAvatar(newAvatar);
|
||||
newAvatar.Url = actor.Uri;
|
||||
@@ -116,7 +116,7 @@ namespace PrivaPub.Services.ClientToServer.Private
|
||||
avatar.Fields = form.Fields ?? new();
|
||||
avatar.PersonalNote = form.PersonalNote;
|
||||
if (form.Settings != default)
|
||||
avatar.Settings = ToSettings(form.Settings);
|
||||
Apply(avatar.Settings ??= new(), form.Settings);
|
||||
avatar.UpdatedAt = DateTime.UtcNow;
|
||||
await DB.Default.SaveAsync(avatar);
|
||||
await _outbox.PublishProfile(_localActors.FromAvatar(avatar), default);
|
||||
@@ -158,18 +158,20 @@ namespace PrivaPub.Services.ClientToServer.Private
|
||||
.Match(ru => !ru.DeletedAt.HasValue && !ru.IsBanned)
|
||||
.ExecuteAnyAsync();
|
||||
|
||||
static AvatarSettings ToSettings(ViewAvatarSettings settings) => new()
|
||||
// The client API carries only the theme; the rest (discoverable, locked, quote policy…) is set through the Mastodon API
|
||||
// and must survive an edit made here.
|
||||
static void Apply(AvatarSettings target, ViewAvatarSettings settings)
|
||||
{
|
||||
IsDefault = settings.IsDefault,
|
||||
DarkThemeIndexColour = settings.DarkThemeIndexColour,
|
||||
IconsThemeIndexColour = settings.IconsThemeIndexColour,
|
||||
LanguageCode = settings.LanguageCode,
|
||||
LightThemeIndexColour = settings.LightThemeIndexColour,
|
||||
PreferSystemTheming = settings.PreferSystemTheming,
|
||||
ThemeIsDarkGray = settings.ThemeIsDarkGray,
|
||||
ThemeIsDarkMode = settings.ThemeIsDarkMode,
|
||||
ThemeIsLightGray = settings.ThemeIsLightGray
|
||||
};
|
||||
target.IsDefault = settings.IsDefault;
|
||||
target.DarkThemeIndexColour = settings.DarkThemeIndexColour;
|
||||
target.IconsThemeIndexColour = settings.IconsThemeIndexColour;
|
||||
target.LanguageCode = settings.LanguageCode;
|
||||
target.LightThemeIndexColour = settings.LightThemeIndexColour;
|
||||
target.PreferSystemTheming = settings.PreferSystemTheming;
|
||||
target.ThemeIsDarkGray = settings.ThemeIsDarkGray;
|
||||
target.ThemeIsDarkMode = settings.ThemeIsDarkMode;
|
||||
target.ThemeIsLightGray = settings.ThemeIsLightGray;
|
||||
}
|
||||
|
||||
ViewAvatar ToView(Avatar avatar) => new()
|
||||
{
|
||||
|
||||
@@ -133,14 +133,18 @@ namespace PrivaPub.Web.Pages
|
||||
readonly Microsoft.Extensions.Options.IOptionsMonitor<Infrastructure.Statistics.StatisticsOptions> _options;
|
||||
readonly Microsoft.Extensions.Options.IOptionsMonitor<Models.AppConfiguration> _app;
|
||||
|
||||
readonly Infrastructure.Geo.IGeoLocator _geo;
|
||||
|
||||
public StargazingModel(Microsoft.Extensions.Options.IOptionsMonitor<Infrastructure.Statistics.StatisticsOptions> options,
|
||||
Microsoft.Extensions.Options.IOptionsMonitor<Models.AppConfiguration> app)
|
||||
Microsoft.Extensions.Options.IOptionsMonitor<Models.AppConfiguration> app, Infrastructure.Geo.IGeoLocator geo)
|
||||
{
|
||||
_options = options;
|
||||
_app = app;
|
||||
_geo = geo;
|
||||
}
|
||||
|
||||
public bool CrawlerEnabled => _options.CurrentValue.Crawler.Enabled;
|
||||
public string GeoSource => _geo.Source;
|
||||
public string UserAgent => Federation.Crawler.Stargazer.UserAgent(_app.CurrentValue.BackendBaseAddress);
|
||||
|
||||
public void OnGet() => Harden();
|
||||
|
||||
@@ -15,6 +15,15 @@
|
||||
<p>A server we exchange activities with is described once a week, from its public NodeInfo and, when it has one, its
|
||||
Mastodon instance API. Its location comes from the address we reached, looked up in an offline database: only the
|
||||
country is shown publicly for small servers, and only the CDN for servers behind one.</p>
|
||||
@if (Model.GeoSource is { } geo)
|
||||
{
|
||||
<p>Locations come from <a href="https://db-ip.com" rel="nofollow noopener">@geo</a>, licensed under
|
||||
<a href="https://creativecommons.org/licenses/by/4.0/" rel="nofollow noopener">CC BY 4.0</a>.</p>
|
||||
}
|
||||
else
|
||||
{
|
||||
<p>No location database is loaded yet, so servers are not located.</p>
|
||||
}
|
||||
</article>
|
||||
<article>
|
||||
<h2>The crawler</h2>
|
||||
|
||||
@@ -2,6 +2,26 @@
|
||||
"Media": {
|
||||
"Root": "/var/lib/privapub/media"
|
||||
},
|
||||
"Registrations": {
|
||||
"Mode": "Invitations"
|
||||
},
|
||||
"Statistics": {
|
||||
"Crawler": {
|
||||
"Enabled": true,
|
||||
"Seeds": [
|
||||
"mastodon.social",
|
||||
"fosstodon.org",
|
||||
"mas.to",
|
||||
"lemmy.world",
|
||||
"lemmy.ml",
|
||||
"misskey.io",
|
||||
"pixelfed.social",
|
||||
"framatube.org",
|
||||
"piefed.social",
|
||||
"bookwyrm.social"
|
||||
]
|
||||
}
|
||||
},
|
||||
"MongoSettings": {
|
||||
"Database": "PrivaPub",
|
||||
"LogsDatabase": "logs",
|
||||
|
||||
@@ -1,23 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fetches the month's DB-IP Lite city and ASN databases (CC BY 4.0, https://db-ip.com) into the directory PrivaPub reads
|
||||
# them from (Statistics:GeoDirectory). The app swaps to new files on its own; a failed download leaves the old ones.
|
||||
set -euo pipefail
|
||||
dir="${GEO_DIR:-/var/lib/privapub/geo}"
|
||||
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
|
||||
this=$(date -u +%Y-%m)
|
||||
last=$(date -u -d "$(date -u +%Y-%m-15) -1 month" +%Y-%m)
|
||||
for kind in city asn; do
|
||||
got=""
|
||||
for month in "$this" "$last"; do
|
||||
if curl -fsS --max-time 900 -o "$tmp/$kind.gz" "https://download.db-ip.com/free/dbip-$kind-lite-$month.mmdb.gz"; then
|
||||
got=$month; break
|
||||
fi
|
||||
done
|
||||
[ -n "$got" ] || { echo "no $kind database for $this or $last" >&2; exit 1; }
|
||||
gunzip -t "$tmp/$kind.gz"
|
||||
gunzip -c "$tmp/$kind.gz" > "$tmp/dbip-$kind-lite.mmdb"
|
||||
[ "$(stat -c %s "$tmp/dbip-$kind-lite.mmdb")" -gt 1000000 ] || { echo "the $kind database is too small" >&2; exit 1; }
|
||||
install -m 640 "$tmp/dbip-$kind-lite.mmdb" "$dir/.dbip-$kind-lite.mmdb.new"
|
||||
mv -f "$dir/.dbip-$kind-lite.mmdb.new" "$dir/dbip-$kind-lite.mmdb"
|
||||
echo "$kind: DB-IP Lite $got"
|
||||
done
|
||||
+1
-6
@@ -12,7 +12,7 @@ ACME=/root/.acme.sh/acme.sh
|
||||
echo "== directories"
|
||||
install -d -o "$RUNNER" -g www-data -m 755 /var/www/$HOST
|
||||
install -d -o "$RUNNER" -g "$RUNNER" -m 750 /var/backups/$HOST
|
||||
install -d -o www-data -g www-data -m 750 /var/lib/privapub /var/lib/privapub/mongo /var/lib/privapub/geo
|
||||
install -d -o www-data -g www-data -m 750 /var/lib/privapub /var/lib/privapub/mongo
|
||||
|
||||
echo "== sudoers"
|
||||
SUDOERS=/etc/sudoers.d/$RUNNER
|
||||
@@ -23,15 +23,10 @@ visudo -cf "$SUDOERS"
|
||||
echo "== units"
|
||||
install -m 644 "$SRC/systemd/privapub-mongod.service" /etc/systemd/system/privapub-mongod.service
|
||||
install -m 644 "$SRC/systemd/$UNIT.service" /etc/systemd/system/$UNIT.service
|
||||
install -m 755 "$SRC/max/geo-update.sh" /usr/local/bin/privapub-geo-update
|
||||
install -m 644 "$SRC/systemd/privapub-geo.service" /etc/systemd/system/privapub-geo.service
|
||||
install -m 644 "$SRC/systemd/privapub-geo.timer" /etc/systemd/system/privapub-geo.timer
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now privapub-mongod >/dev/null
|
||||
systemctl enable $UNIT >/dev/null
|
||||
systemctl enable --now privapub-geo.timer >/dev/null
|
||||
systemctl is-active privapub-mongod
|
||||
[ -f /var/lib/privapub/geo/dbip-city-lite.mmdb ] || systemctl start privapub-geo.service || echo "geolocation databases not fetched yet; the timer retries"
|
||||
|
||||
echo "== nginx snippet and bootstrap vhost"
|
||||
install -m 644 "$SRC/nginx/privapub-headers.conf" /etc/nginx/snippets/privapub-headers.conf
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
[Unit]
|
||||
Description=PrivaPub: fetch the DB-IP Lite geolocation databases
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=www-data
|
||||
Group=www-data
|
||||
ExecStart=/usr/local/bin/privapub-geo-update
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
ReadWritePaths=/var/lib/privapub/geo
|
||||
@@ -1,10 +0,0 @@
|
||||
[Unit]
|
||||
Description=PrivaPub: refresh the geolocation databases monthly
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-03 04:00:00
|
||||
RandomizedDelaySec=6h
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
+22
-5
@@ -29,8 +29,11 @@ Written 2026-10-01 from the original 2023 code, the decePubClient UI, a federati
|
||||
- followers-only posts arrived as DMs on Pleroma and Akkoma;
|
||||
- Akkoma's open polls showed as ended and refused votes.
|
||||
|
||||
Still open: installing the geolocation timer on Max (`deploy/max/setup.sh`, as root), and the smoke persona's
|
||||
`PRIVAPUB_SMOKE_TOKEN` secret, without which the deploy skips the signed-in half of its Mastodon API check.
|
||||
Both open items were closed without a root step (owner decisions, 2026-10-04): the server fetches its geolocation
|
||||
databases itself, and the deploy makes and signs in as @thepra.
|
||||
- [ ] Everything on in production (owner decisions 2026-10-04): v1.18.0 self-updating geolocation, @thepra, the crawler
|
||||
on, sign-up by invitation, one registrations switch; then signed audiences with circles for everyone and SecureMode on,
|
||||
account privacy, and one answer everywhere (the mismatch sweep).
|
||||
- [ ] P7 Threads, communities, moderation, the social graph
|
||||
- [ ] P8 Signatures, discovery, the long tail
|
||||
|
||||
@@ -159,10 +162,24 @@ and circles (see Owner decisions).
|
||||
| Local side in statistics | **Only the kind of local actor** (person, group, application), **and only on public and unlisted traffic.** DMs, followers-only and circle traffic are one "private" class, never broken out per server in public. Circles are never named, whether as a kind or as a reason. Fetches of our own documents are counted per day, never per server. |
|
||||
| Reading-driven traffic | **Counted per day, never logged per event:** the media proxy, lookups a client asks for, and the client API per endpoint group (admin only). Client app names are not recorded. |
|
||||
| Describing servers | **Every server we exchange activities with is described weekly**, from its NodeInfo (including the user counts it publishes) and its Mastodon instance API, never its contact account. These requests are unsigned, because they are not ActivityPub documents. Never on read. |
|
||||
| Server locations | **City and network (ASN) from the offline DB-IP Lite databases** (CC BY 4.0, attributed), downloaded monthly outside the app. The location comes from the address we connected to; an inbound sender's address is never recorded, and no address is stored. In public: city and network only for servers reporting at least 10 users and not behind a CDN; the country otherwise; only the CDN's name for CDN-fronted servers. The admin sees everything. |
|
||||
| Crawler | **Off by default** (`Statistics:Crawler:Enabled`). When on, it identifies as `PrivaPub-Stargazer/<version> (+https://privapub.thepra.dev/stargazing)`, where `/stargazing` explains it and how to opt out. It honours robots.txt (an unreachable robots.txt means "keep out") and domain blocks. It visits one server a minute, each at most weekly, and at most 5000 servers. It reads only robots.txt, NodeInfo, the instance API and the peers list, never accounts, posts or directories. Crawled servers stay marked as crawled. |
|
||||
| Server locations | **City and network (ASN) from the offline DB-IP Lite databases** (CC BY 4.0, attributed), downloaded monthly (by the server itself since 2026-10-04). The location comes from the address we connected to; an inbound sender's address is never recorded, and no address is stored. In public: city and network only for servers reporting at least 10 users and not behind a CDN; the country otherwise; only the CDN's name for CDN-fronted servers. The admin sees everything. |
|
||||
| Crawler | **Off by default** (`Statistics:Crawler:Enabled`); **on in production since 2026-10-04**, see below. When on, it identifies as `PrivaPub-Stargazer/<version> (+https://privapub.thepra.dev/stargazing)`, where `/stargazing` explains it and how to opt out. It honours robots.txt (an unreachable robots.txt means "keep out") and domain blocks. It visits one server a minute, each at most weekly, and at most 5000 servers. It reads only robots.txt, NodeInfo, the instance API and the peers list, never accounts, posts or directories. Crawled servers stay marked as crawled. |
|
||||
| A remote account deletes itself | **Its posts are kept but hidden everywhere** (`Post.AuthorGone`): from timelines, profiles, search and lookups by id. Its follows and timeline rows go, as before. |
|
||||
| Signed-in smoke check in production | **An undiscoverable persona**, whose read-only token is the Gitea secret `PRIVAPUB_SMOKE_TOKEN`; the deploy checks `verify_credentials`, home and notifications with it. The owner creates both. |
|
||||
| Signed-in smoke check in production | **An undiscoverable persona**, the deploy checks `verify_credentials`, home and notifications with it. *Superseded 2026-10-04: the deploy makes and keeps the persona itself, below.* |
|
||||
|
||||
### Owner decisions on running everything in production (2026-10-04)
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| What runs in production | **Everything that is built is on and checked by the deploy**, and nothing waits on a person running a command. |
|
||||
| Geolocation | **The server fetches DB-IP Lite itself** (`GeoUpdater`): it checks daily, installs a new month's databases once they are published, refuses a file that does not open as the right kind of database, and keeps the old one when anything fails. No timer and no root step. The deploy fails if the databases are missing or more than 40 days old. |
|
||||
| Crawler | **On in production**, seeded with a handful of large servers of different kinds (`appsettings.Production.json`); the deploy fails if `/stargazing` does not say it is on. |
|
||||
| Sign-up | **Closed: invitations only.** A group invitation creates an account; open sign-up answers 403. NodeInfo, `/api/v1/instance` and `/api/v2/instance` read the same switch (`Registrations:Mode`), and the deploy fails if they disagree. The first login on a server is made with `PrivaPub admin create-root`. |
|
||||
| Signed-in smoke check | **`@thepra`, undiscoverable, made and kept by the deploy**: `PrivaPub admin smoke thepra` creates or keeps the root `deploy-smoke` and the persona and gives the root a new password on every deploy; the deploy signs in through the real OAuth flow, checks the signed-in API and revokes its token. No secret is stored. |
|
||||
| Signed fetches (SecureMode) | **On in production** once the pasture passes with it on (Phase 2 of the 2026-10-04 plan). |
|
||||
| Circles on Mastodon and GoToSocial | **Each member's copy names that member** in `cc`; a member's refetch names the member, an instance actor's refetch the members on its server. Nothing new is revealed to anyone outside the circle. |
|
||||
| What circles and located posts reveal | **Unchanged**: circles still answer WebFinger, and circle and located posts still count in a persona's post count, "a good balance for the fediverse to work". |
|
||||
| Public `/stargazing` statistics | **Later**, as decided on 2026-10-03; the crawler and the admin API keep collecting meanwhile. |
|
||||
|
||||
## Libraries (researched; no maintained .NET ActivityPub library exists, so Letterbook and Iceshrimp.NET both wrote their own)
|
||||
|
||||
|
||||
@@ -26,6 +26,8 @@
|
||||
},
|
||||
"Media": { "Root": "/tmp/privapub-media" },
|
||||
"RateLimits": { "AccountsPerMinute": 1000 },
|
||||
"Registrations": { "Mode": "Open" },
|
||||
"Statistics": { "Geo": { "AutoUpdate": false } },
|
||||
"Kestrel": { "Endpoints": { "Http": { "Url": "http://0.0.0.0:80", "Protocols": "Http1AndHttp2" } } },
|
||||
"Serilog": {
|
||||
"MinimumLevel": { "Default": "Information", "Override": { "Microsoft": "Warning", "System": "Warning" } },
|
||||
|
||||
@@ -30,37 +30,14 @@ privapub_root() {
|
||||
echo "$root" | j "print(d['token'])"
|
||||
}
|
||||
|
||||
# privapub_token <persona>: creates the persona under the pasture root if needed and returns a Mastodon token for it.
|
||||
# privapub_token <persona>: creates the persona under the pasture root if needed and returns a Mastodon token for it,
|
||||
# through the same OAuth flow the deploy's smoke check uses (tools/smoke/oauth.sh).
|
||||
privapub_token() {
|
||||
local persona=$1 jwt cid cs q xt form code
|
||||
local persona=$1 jwt
|
||||
jwt=$(privapub_root)
|
||||
curl -s -o /dev/null -X POST $P/clientapi/avatar/private/insert -H 'Content-Type: application/json' -H "Authorization: Bearer $jwt" \
|
||||
-d "{\"userName\":\"$persona\",\"name\":\"$persona of PrivaPub\",\"biography\":\"testing federation\"}"
|
||||
local app; app=$(curl -s -X POST $P/api/v1/apps -d 'client_name=pasture&redirect_uris=urn:ietf:wg:oauth:2.0:oob&scopes=read+write+follow')
|
||||
cid=$(echo "$app" | j "print(d['client_id'])"); cs=$(echo "$app" | j "print(d['client_secret'])")
|
||||
q="client_id=$cid&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code&scope=read+write+follow"
|
||||
local jar="$work/jar-$persona"
|
||||
xt=$(curl -s -c "$jar" -b "$jar" "$P/oauth/login?returnUrl=/oauth/authorize?$q" | grep -o 'name="__RequestVerificationToken" type="hidden" value="[^"]*"' | sed 's/.*value="//;s/"//')
|
||||
curl -s -o /dev/null -c "$jar" -b "$jar" -X POST $P/oauth/login --data-urlencode "returnUrl=/oauth/authorize?$q" --data-urlencode "__RequestVerificationToken=$xt" \
|
||||
--data-urlencode "userName=$ROOT_USER" --data-urlencode "password=$ROOT_PASS"
|
||||
curl -s -c "$jar" -b "$jar" "$P/oauth/authorize?$q&signed_in=1" > "$work/choose-$persona.html"
|
||||
form=$(python3 - "$work/choose-$persona.html" "$persona" <<'PY'
|
||||
import re,sys,urllib.parse,html
|
||||
s=open(sys.argv[1]).read()
|
||||
pairs=[(k,html.unescape(v)) for k,v in re.findall(r'<input type="hidden" name="([^"]*)" value="([^"]*)"',s)]
|
||||
blocks=re.findall(r'<label[^>]*>(.*?)</label>',s,re.S)
|
||||
avatar=None
|
||||
for b in blocks:
|
||||
if '@'+sys.argv[2]+'@' in b or '@'+sys.argv[2]+'<' in b or '>'+sys.argv[2]+'<' in b:
|
||||
m=re.search(r'name="avatarId" value="([^"]*)"',b)
|
||||
if m: avatar=m.group(1)
|
||||
if avatar is None:
|
||||
avatar=re.findall(r'name="avatarId" value="([^"]*)"',s)[0]
|
||||
print(urllib.parse.urlencode(pairs+[("avatarId",avatar),("decision","allow")]))
|
||||
PY
|
||||
)
|
||||
code=$(curl -s -c "$jar" -b "$jar" -X POST $P/oauth/authorize --data "$form" | grep -o '<code>[^<]*</code>' | sed 's/<[^>]*>//g')
|
||||
curl -s -X POST $P/oauth/token -d "grant_type=authorization_code&code=$code&client_id=$cid&client_secret=$cs&redirect_uri=urn:ietf:wg:oauth:2.0:oob" | j "print(d['access_token'])"
|
||||
"$here/../smoke/oauth.sh" "$P" "$ROOT_USER" "$ROOT_PASS" "$persona" "read write follow" | cut -d' ' -f1
|
||||
}
|
||||
|
||||
# stats_check <host> <software>: the admin statistics name the peer's software and count traffic both ways.
|
||||
|
||||
Executable
+42
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env bash
|
||||
# Gets a Mastodon API token for one persona through PrivaPub's real OAuth code flow, as a client would: register an app,
|
||||
# sign in with the root's password at /oauth/login, choose the persona at /oauth/authorize, read the out-of-band code and
|
||||
# exchange it. Prints "<access token> <client id> <client secret>" so the caller can revoke the token afterwards.
|
||||
# usage: tools/smoke/oauth.sh <base url> <root login> <root password> <persona> [scopes]
|
||||
set -euo pipefail
|
||||
BASE="${1:?base url}"; LOGIN="${2:?root login}"; PASSWORD="${3:?root password}"; PERSONA="${4:?persona}"; SCOPES="${5:-read}"
|
||||
jar=$(mktemp -d); trap 'rm -rf "$jar"' EXIT
|
||||
json() { python3 -c "import sys,json; d=json.load(sys.stdin); print($1)"; }
|
||||
|
||||
app=$(curl -fsS -X POST "$BASE/api/v1/apps" --data-urlencode client_name=privapub-oauth \
|
||||
--data-urlencode redirect_uris=urn:ietf:wg:oauth:2.0:oob --data-urlencode "scopes=$SCOPES")
|
||||
cid=$(echo "$app" | json "d['client_id']"); cs=$(echo "$app" | json "d['client_secret']")
|
||||
scope_q=$(python3 -c "import sys,urllib.parse; print(urllib.parse.quote(sys.argv[1]))" "$SCOPES")
|
||||
q="client_id=$cid&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code&scope=$scope_q"
|
||||
|
||||
xt=$(curl -fsS -c "$jar/c" -b "$jar/c" "$BASE/oauth/login?returnUrl=/oauth/authorize?$q" \
|
||||
| grep -o 'name="__RequestVerificationToken" type="hidden" value="[^"]*"' | sed 's/.*value="//;s/"//')
|
||||
curl -sS -o /dev/null -c "$jar/c" -b "$jar/c" -X POST "$BASE/oauth/login" --data-urlencode "returnUrl=/oauth/authorize?$q" \
|
||||
--data-urlencode "__RequestVerificationToken=$xt" --data-urlencode "userName=$LOGIN" --data-urlencode "password=$PASSWORD"
|
||||
curl -fsS -c "$jar/c" -b "$jar/c" "$BASE/oauth/authorize?$q&signed_in=1" > "$jar/choose.html"
|
||||
form=$(python3 - "$jar/choose.html" "$PERSONA" <<'PY'
|
||||
import re, sys, urllib.parse, html
|
||||
page = open(sys.argv[1]).read()
|
||||
pairs = [(k, html.unescape(v)) for k, v in re.findall(r'<input type="hidden" name="([^"]*)" value="([^"]*)"', page)]
|
||||
persona = None
|
||||
for block in re.findall(r'<label[^>]*>(.*?)</label>', page, re.S):
|
||||
if '@' + sys.argv[2] + '@' in block or '@' + sys.argv[2] + '<' in block or '>' + sys.argv[2] + '<' in block:
|
||||
found = re.search(r'name="avatarId" value="([^"]*)"', block)
|
||||
if found:
|
||||
persona = found.group(1)
|
||||
if persona is None:
|
||||
sys.exit("persona '%s' is not offered at /oauth/authorize" % sys.argv[2])
|
||||
print(urllib.parse.urlencode(pairs + [("avatarId", persona), ("decision", "allow")]))
|
||||
PY
|
||||
)
|
||||
code=$(curl -fsS -c "$jar/c" -b "$jar/c" -X POST "$BASE/oauth/authorize" --data "$form" | grep -o '<code>[^<]*</code>' | sed 's/<[^>]*>//g')
|
||||
[ -n "$code" ] || { echo "no authorization code" >&2; exit 1; }
|
||||
token=$(curl -fsS -X POST "$BASE/oauth/token" --data-urlencode grant_type=authorization_code --data-urlencode "code=$code" \
|
||||
--data-urlencode "client_id=$cid" --data-urlencode "client_secret=$cs" --data-urlencode redirect_uri=urn:ietf:wg:oauth:2.0:oob \
|
||||
| json "d['access_token']")
|
||||
echo "$token $cid $cs"
|
||||
Reference in new issue
Block a user