Everything on, phase 1: geolocation fetches itself, the deploy signs in as @thepra, the crawler is on, sign-up by invitation
Owner decisions (2026-10-04, recorded in docs/ROADMAP.md): production runs everything that is built, and nothing waits
on a person running a command.
- Geolocation updates itself. GeoUpdater, a hosted service, checks daily whether each DB-IP Lite database was built this
month. If not, it fetches this month's, or last month's early in the month. It installs a file only once it opens as
the right kind of database, then swaps it in atomically, and the locator reloads at once. Lookups now run under the
lock, so a reload can no longer dispose a reader mid-lookup. The systemd timer, its script and their setup.sh lines
are gone: the root step they needed never happened, and none is needed now. /stargazing names the database in use.
- The admin CLI runs after the app is built, with every service and nothing started.
- `create-root <login> [--admin]` takes the password on stdin; it is how the first login is made while sign-up is
closed.
- `smoke <persona>` keeps the root `deploy-smoke` and an undiscoverable persona, and gives the root a new password
on every run.
- The deploy signs in as @thepra. It runs the CLI, gets a token through the real OAuth flow (tools/smoke/oauth.sh,
moved out of the pasture's privapub_token, which now uses it), checks the signed-in API and that @thepra is
undiscoverable, then revokes the token. PRIVAPUB_SMOKE_TOKEN is gone.
- The deploy also fails when:
- NodeInfo and the instance API disagree about registrations;
- /stargazing does not say the crawler is on;
- the geolocation databases are missing or more than 40 days old.
- The crawler is on in production, seeded with ten large servers of different kinds. FEDERATION.md now describes it
and how to opt out.
- One registrations switch (Registrations:Mode, default Invitations; Open in tests and the pasture). It is read by
open sign-up (403 when closed), NodeInfo `openRegistrations`, and v1 and v2 of the instance API, so they can no longer
disagree. Before, NodeInfo said open and the instance API said closed. Group invitations always work, so
invites_enabled is true.
- A persona edit through /clientapi no longer resets what the Mastodon API set (discoverable, locked, quote policy…):
the theme is merged into the settings instead of replacing them.
650 tests pass. The deploy's smoke step was rehearsed against the pasture's PrivaPub.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
This commit is contained in:
1 parent
cd5eb25948
commit
5f56681c01
33 files changed
+909
-157
No files matched your search
@@ -1,11 +1,13 @@
|
||||
using Microsoft.AspNetCore.Authorization;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Api.Mastodon.Infrastructure;
|
||||
using PrivaPub.Domain.Privacy;
|
||||
using PrivaPub.Federation.Actors;
|
||||
using PrivaPub.Infrastructure;
|
||||
using PrivaPub.Models.User;
|
||||
|
||||
using PostEntity = PrivaPub.Models.Post.Post;
|
||||
@@ -18,12 +20,16 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
const string Description = "A small ActivityPub server where one private login keeps several unlinkable public personas.";
|
||||
|
||||
readonly ILocalActorService _localActors;
|
||||
readonly IOptionsMonitor<RegistrationOptions> _registrations;
|
||||
|
||||
public InstanceController(ILocalActorService localActors)
|
||||
public InstanceController(ILocalActorService localActors, IOptionsMonitor<RegistrationOptions> registrations)
|
||||
{
|
||||
_localActors = localActors;
|
||||
_registrations = registrations;
|
||||
}
|
||||
|
||||
bool Open => _registrations.CurrentValue.IsOpen;
|
||||
|
||||
string Domain => new Uri(_localActors.BaseAddress).Authority;
|
||||
|
||||
static object Statuses => new { max_characters = 5000, max_media_attachments = 4, characters_reserved_per_url = 23 };
|
||||
@@ -58,9 +64,9 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
stats = new { user_count = users, status_count = statuses, domain_count = domains },
|
||||
thumbnail = $"{_localActors.BaseAddress}/media/missing-header.png",
|
||||
languages = new[] { "en" },
|
||||
registrations = false,
|
||||
registrations = Open,
|
||||
approval_required = false,
|
||||
invites_enabled = false,
|
||||
invites_enabled = true,//a group invitation always creates an account
|
||||
configuration = new { accounts = new { max_featured_tags = 0 }, statuses = Statuses, media_attachments = MediaAttachments, polls = Polls },
|
||||
contact_account = default(object),
|
||||
rules = Array.Empty<object>()
|
||||
@@ -88,7 +94,7 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
polls = Polls,
|
||||
translation = new { enabled = false }
|
||||
},
|
||||
registrations = new { enabled = false, approval_required = false, message = default(string) },
|
||||
registrations = new { enabled = Open, approval_required = false, message = Open ? default : RegistrationOptions.ClosedMessage },
|
||||
contact = new { email = string.Empty, account = default(object) },
|
||||
rules = Array.Empty<object>()
|
||||
});
|
||||
|
||||
@@ -34,6 +34,7 @@ namespace PrivaPub.Controllers.ClientToServer
|
||||
readonly ILocalActorService LocalActors;
|
||||
readonly AuthTokenManager AuthTokenManager;
|
||||
readonly IOptionsMonitor<AppConfiguration> AppConfiguration;
|
||||
readonly IOptionsMonitor<RegistrationOptions> Registrations;
|
||||
readonly ILogger<RootUserController> Logger;
|
||||
readonly IStringLocalizer Localizer;
|
||||
|
||||
@@ -43,9 +44,11 @@ namespace PrivaPub.Controllers.ClientToServer
|
||||
ILocalActorService localActors,
|
||||
AuthTokenManager authTokenManager,
|
||||
IOptionsMonitor<AppConfiguration> appConfiguration,
|
||||
IOptionsMonitor<RegistrationOptions> registrations,
|
||||
IStringLocalizer<GenericRes> localizer,
|
||||
ILogger<RootUserController> logger)
|
||||
{
|
||||
Registrations = registrations;
|
||||
UsersService = usersService;
|
||||
GroupUsersService = groupUsersService;
|
||||
AvatarUsersService = avatarUsersService;
|
||||
@@ -63,6 +66,8 @@ namespace PrivaPub.Controllers.ClientToServer
|
||||
{
|
||||
if (User.Identity?.IsAuthenticated ?? false) return Redirect("/");
|
||||
var result = new WebResult();
|
||||
if (!Registrations.CurrentValue.IsOpen)
|
||||
return StatusCode(StatusCodes.Status403Forbidden, result.Invalidate(Localizer[RegistrationOptions.ClosedMessage], StatusCodes.Status403Forbidden));
|
||||
if (!ModelState.IsValid)
|
||||
return BadRequest(result.Invalidate(Localizer["Invalid model."]));
|
||||
try
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
using MongoDB.Entities;
|
||||
|
||||
@@ -12,6 +13,7 @@ using System.Text.Json.Nodes;
|
||||
using PostEntity = PrivaPub.Models.Post.Post;
|
||||
using PrivaPub.Domain.Privacy;
|
||||
using PrivaPub.Federation.Actors;
|
||||
using PrivaPub.Infrastructure;
|
||||
|
||||
namespace PrivaPub.Federation.Controllers
|
||||
{
|
||||
@@ -20,9 +22,11 @@ namespace PrivaPub.Federation.Controllers
|
||||
{
|
||||
readonly ILocalActorService _localActors;
|
||||
readonly DbEntities _dbEntities;
|
||||
readonly IOptionsMonitor<RegistrationOptions> _registrations;
|
||||
|
||||
public WellKnownController(ILocalActorService localActors, DbEntities dbEntities)
|
||||
public WellKnownController(ILocalActorService localActors, DbEntities dbEntities, IOptionsMonitor<RegistrationOptions> registrations)
|
||||
{
|
||||
_registrations = registrations;
|
||||
_localActors = localActors;
|
||||
_dbEntities = dbEntities;
|
||||
}
|
||||
@@ -102,7 +106,7 @@ namespace PrivaPub.Federation.Controllers
|
||||
["software"] = software,
|
||||
["protocols"] = new JsonArray("activitypub"),
|
||||
["services"] = new JsonObject { ["inbound"] = new JsonArray(), ["outbound"] = new JsonArray() },
|
||||
["openRegistrations"] = true,
|
||||
["openRegistrations"] = _registrations.CurrentValue.IsOpen,
|
||||
["usage"] = new JsonObject
|
||||
{
|
||||
["users"] = new JsonObject { ["total"] = users },
|
||||
|
||||
@@ -1,22 +1,49 @@
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.ClientModels;
|
||||
using PrivaPub.ClientModels.User;
|
||||
using PrivaPub.ClientModels.User.Avatar;
|
||||
using PrivaPub.Models.User;
|
||||
using PrivaPub.Services;
|
||||
using PrivaPub.Services.ClientToServer.Private;
|
||||
using PrivaPub.StaticServices;
|
||||
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
using System.Security.Cryptography;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Cli
|
||||
{
|
||||
// `PrivaPub admin ...` runs with the whole server built but not started: no Kestrel, no hosted services.
|
||||
public static class AdminCommands
|
||||
{
|
||||
const string Usage = "usage: PrivaPub admin promote|demote <root username>";
|
||||
public const string SmokeLogin = "deploy-smoke";
|
||||
|
||||
public static async Task<int> Run(string[] args)
|
||||
const string Usage = """
|
||||
usage: PrivaPub admin promote|demote <root>
|
||||
PrivaPub admin create-root <login> [--admin] the password is read from standard input
|
||||
PrivaPub admin smoke <persona> prints "<login> <password>" for the deploy's signed-in check
|
||||
""";
|
||||
|
||||
public static async Task<int> Run(string[] args, IServiceProvider services, TextReader input = default, TextWriter output = default)
|
||||
{
|
||||
if (args is not [("promote" or "demote") and var verb, var userName])
|
||||
input ??= Console.In;
|
||||
output ??= Console.Out;
|
||||
switch (args)
|
||||
{
|
||||
Console.Error.WriteLine(Usage);
|
||||
return 2;
|
||||
case [("promote" or "demote") and var verb, var userName]:
|
||||
return await Promote(verb == "promote", userName, output);
|
||||
case ["create-root", var login, .. var flags] when flags.All(f => f == "--admin"):
|
||||
return await CreateRoot(services, login, input.ReadLine(), flags.Contains("--admin"), output);
|
||||
case ["smoke", var persona]:
|
||||
return await Smoke(services, persona, output);
|
||||
default:
|
||||
Console.Error.WriteLine(Usage);
|
||||
return 2;
|
||||
}
|
||||
}
|
||||
|
||||
static async Task<int> Promote(bool promote, string userName, TextWriter output)
|
||||
{
|
||||
userName = userName.ToLowerInvariant();
|
||||
var user = await DB.Default.Find<RootUser>().Match(u => u.UserName == userName).ExecuteFirstAsync();
|
||||
if (user == default)
|
||||
@@ -26,14 +53,94 @@ namespace PrivaPub.Infrastructure.Cli
|
||||
}
|
||||
|
||||
user.Policies.RemoveAll(p => p is Policies.IsAdmin or Policies.IsModerator);
|
||||
if (verb == "promote")
|
||||
if (promote)
|
||||
user.Policies.AddRange(new[] { Policies.IsAdmin, Policies.IsModerator });
|
||||
if (!user.Policies.Contains(Policies.IsUser))
|
||||
user.Policies.Add(Policies.IsUser);
|
||||
user.UpdatedAt = DateTime.UtcNow;
|
||||
await DB.Default.SaveAsync(user);
|
||||
|
||||
Console.WriteLine($"{userName}: {string.Join(", ", user.Policies)}");
|
||||
output.WriteLine($"{userName}: {string.Join(", ", user.Policies)}");
|
||||
return 0;
|
||||
}
|
||||
|
||||
// With sign-up closed this is how the first root is made; group invitations make the rest.
|
||||
static async Task<int> CreateRoot(IServiceProvider services, string login, string password, bool admin, TextWriter output)
|
||||
{
|
||||
var form = new LoginForm { UserName = login, Password = password?.Trim() };
|
||||
var problems = new List<ValidationResult>();
|
||||
if (!Validator.TryValidateObject(form, new ValidationContext(form), problems, validateAllProperties: true))
|
||||
{
|
||||
Console.Error.WriteLine(string.Join(Environment.NewLine, problems.Select(p => p.ErrorMessage)));
|
||||
return 1;
|
||||
}
|
||||
var created = await services.GetRequiredService<IRootUsersService>().SignUpAsync(form);
|
||||
if (!created.IsValid)
|
||||
{
|
||||
Console.Error.WriteLine(created.ErrorMessage);
|
||||
return 1;
|
||||
}
|
||||
return admin ? await Promote(true, login, output) : await Report(login, output);
|
||||
}
|
||||
|
||||
static Task<int> Report(string login, TextWriter output)
|
||||
{
|
||||
output.WriteLine($"{login.ToLowerInvariant()}: created");
|
||||
return Task.FromResult(0);
|
||||
}
|
||||
|
||||
// The deploy's signed-in smoke check: a root nobody signs in to by hand, owning one undiscoverable persona. Every run
|
||||
// sets a new password and prints it, so no secret is kept anywhere and nothing waits on a person.
|
||||
static async Task<int> Smoke(IServiceProvider services, string personaName, TextWriter output)
|
||||
{
|
||||
personaName = personaName.ToLowerInvariant();
|
||||
var password = "Smoke-x" + Convert.ToHexStringLower(RandomNumberGenerator.GetBytes(24));
|
||||
var root = await DB.Default.Find<RootUser>().Match(u => u.UserName == SmokeLogin).ExecuteFirstAsync();
|
||||
if (root == default)
|
||||
{
|
||||
var created = await services.GetRequiredService<IRootUsersService>().SignUpAsync(new LoginForm { UserName = SmokeLogin, Password = password });
|
||||
if (!created.IsValid)
|
||||
{
|
||||
Console.Error.WriteLine(created.ErrorMessage);
|
||||
return 1;
|
||||
}
|
||||
root = await DB.Default.Find<RootUser>().Match(u => u.UserName == SmokeLogin).ExecuteFirstAsync();
|
||||
}
|
||||
else if (root.DeletedAt.HasValue || root.IsBanned)
|
||||
{
|
||||
Console.Error.WriteLine($"the root '{SmokeLogin}' is deleted or banned");
|
||||
return 1;
|
||||
}
|
||||
else
|
||||
await DB.Default.Update<RootUser>().MatchID(root.ID)
|
||||
.Modify(u => u.HashedPassword, services.GetRequiredService<IPasswordHasher>().Hash(password))
|
||||
.Modify(u => u.UpdatedAt, DateTime.UtcNow)
|
||||
.ExecuteAsync();
|
||||
|
||||
var owned = (await DB.Default.Find<RootToAvatar>().Match(ra => ra.RootId == root.ID).ExecuteAsync()).Select(ra => ra.AvatarId).ToList();
|
||||
var persona = await DB.Default.Find<Avatar>().Match(a => owned.Contains(a.ID) && a.UserName == personaName && !a.DeletionAt.HasValue).ExecuteFirstAsync();
|
||||
if (persona == default)
|
||||
{
|
||||
var inserted = await services.GetRequiredService<IPrivateAvatarUsersService>().InsertAvatar(new InsertAvatarForm
|
||||
{
|
||||
RootId = root.ID,
|
||||
UserName = personaName,
|
||||
Name = personaName,
|
||||
Biography = "The deploy signs in here to check that the Mastodon API works for a signed-in persona."
|
||||
});
|
||||
if (!inserted.IsValid)
|
||||
{
|
||||
Console.Error.WriteLine(inserted.ErrorMessage);
|
||||
return 1;
|
||||
}
|
||||
persona = await DB.Default.Find<Avatar>().MatchID(((ViewAvatar)inserted.Data).Id).ExecuteFirstAsync();
|
||||
}
|
||||
await DB.Default.Update<Avatar>().MatchID(persona.ID)
|
||||
.Modify(a => a.Settings.IsDiscoverable, false)
|
||||
.Modify(a => a.Settings.IsIndexable, false)
|
||||
.ExecuteAsync();
|
||||
|
||||
output.WriteLine($"{SmokeLogin} {password}");
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -25,8 +25,8 @@ namespace PrivaPub.Infrastructure.Geo
|
||||
|
||||
public sealed class DbIpLocator : IGeoLocator, IDisposable
|
||||
{
|
||||
public const string CityFile = "dbip-city-lite.mmdb";
|
||||
public const string AsnFile = "dbip-asn-lite.mmdb";
|
||||
public static readonly string CityFile = GeoUpdater.FileOf("city");
|
||||
public static readonly string AsnFile = GeoUpdater.FileOf("asn");
|
||||
static readonly TimeSpan CheckInterval = TimeSpan.FromMinutes(10);
|
||||
|
||||
readonly IOptionsMonitor<StatisticsOptions> _options;
|
||||
@@ -60,18 +60,17 @@ namespace PrivaPub.Infrastructure.Geo
|
||||
if (address == default || !IpRangeGuard.IsPublic(address))
|
||||
return default;
|
||||
Refresh();
|
||||
Reader city, asn;
|
||||
lock (_lock)
|
||||
{
|
||||
city = _city;
|
||||
asn = _asn;
|
||||
}
|
||||
if (city == default && asn == default)
|
||||
return default;
|
||||
Dictionary<string, object> place, network;
|
||||
try
|
||||
{
|
||||
var place = city?.Find<Dictionary<string, object>>(address);
|
||||
var network = asn?.Find<Dictionary<string, object>>(address);
|
||||
// under the lock, so a reload cannot dispose a reader in the middle of a lookup; lookups are rare
|
||||
lock (_lock)
|
||||
{
|
||||
if (_city == default && _asn == default)
|
||||
return default;
|
||||
place = _city?.Find<Dictionary<string, object>>(address);
|
||||
network = _asn?.Find<Dictionary<string, object>>(address);
|
||||
}
|
||||
var location = Section(place, "location");
|
||||
return new GeoFix(
|
||||
Text(Section(place, "country"), "iso_code"),
|
||||
@@ -88,6 +87,14 @@ namespace PrivaPub.Infrastructure.Geo
|
||||
}
|
||||
}
|
||||
|
||||
// Looks at the files again now instead of within the next ten minutes: GeoUpdater has just replaced one.
|
||||
public void Reload()
|
||||
{
|
||||
lock (_lock)
|
||||
_checkedAt = DateTime.MinValue;
|
||||
Refresh();
|
||||
}
|
||||
|
||||
void Refresh()
|
||||
{
|
||||
if (DateTime.UtcNow - _checkedAt < CheckInterval)
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
using MaxMind.Db;
|
||||
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
using PrivaPub.Infrastructure.Statistics;
|
||||
|
||||
using System.IO.Compression;
|
||||
using System.Net;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Geo
|
||||
{
|
||||
// Keeps the DB-IP Lite databases current by itself, so a deploy needs no timer and no root: once a day it checks
|
||||
// whether each database was built this month and, if not, fetches this month's (or, early in the month, last month's),
|
||||
// checks it opens as the right kind of database and swaps it in. This and SMTP are the server's only traffic that is not
|
||||
// federation, which is why it has its own client instead of IFederationHttp.
|
||||
public sealed class GeoUpdater : BackgroundService
|
||||
{
|
||||
public const string ClientName = "geo";
|
||||
static readonly string[] Kinds = { "city", "asn" };
|
||||
static readonly TimeSpan FirstWait = TimeSpan.FromSeconds(30);
|
||||
static readonly TimeSpan Interval = TimeSpan.FromHours(24);
|
||||
|
||||
readonly IHttpClientFactory _http;
|
||||
readonly IOptionsMonitor<StatisticsOptions> _options;
|
||||
readonly IGeoLocator _locator;
|
||||
readonly ILogger<GeoUpdater> _logger;
|
||||
|
||||
public GeoUpdater(IHttpClientFactory http, IOptionsMonitor<StatisticsOptions> options, IGeoLocator locator, ILogger<GeoUpdater> logger)
|
||||
{
|
||||
_http = http;
|
||||
_options = options;
|
||||
_locator = locator;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
public static string FileOf(string kind) => $"dbip-{kind}-lite.mmdb";
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(FirstWait, stoppingToken);
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
if (_options.CurrentValue.Geo.AutoUpdate)
|
||||
await Update(DateTime.UtcNow, stoppingToken);
|
||||
await Task.Delay(Interval + TimeSpan.FromMinutes(Random.Shared.Next(60)), stoppingToken);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
}
|
||||
}
|
||||
|
||||
// The number of databases replaced. A failure keeps the database already there and is tried again the next day.
|
||||
public async Task<int> Update(DateTime now, CancellationToken token)
|
||||
{
|
||||
var options = _options.CurrentValue;
|
||||
var replaced = 0;
|
||||
try
|
||||
{
|
||||
Directory.CreateDirectory(options.GeoDirectory);
|
||||
}
|
||||
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
_logger.LogWarning(ex, "Cannot create {Directory}; servers are not located", options.GeoDirectory);
|
||||
return 0;
|
||||
}
|
||||
foreach (var kind in Kinds)
|
||||
{
|
||||
var path = Path.Combine(options.GeoDirectory, FileOf(kind));
|
||||
var built = BuiltMonth(path);
|
||||
foreach (var month in new[] { Month(now), Month(now).AddMonths(-1) })
|
||||
{
|
||||
if (built >= month)
|
||||
break;
|
||||
try
|
||||
{
|
||||
if (await Install(kind, month, path, options, token))
|
||||
{
|
||||
replaced++;
|
||||
_logger.LogInformation("Installed DB-IP Lite {Kind} {Month:yyyy-MM}", kind, month);
|
||||
break;
|
||||
}
|
||||
}
|
||||
catch (Exception ex) when (ex is HttpRequestException or IOException or InvalidDatabaseException or InvalidDataException
|
||||
or TaskCanceledException && !token.IsCancellationRequested)
|
||||
{
|
||||
_logger.LogWarning(ex, "Could not install DB-IP Lite {Kind} {Month:yyyy-MM}", kind, month);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (replaced > 0 && _locator is DbIpLocator locator)
|
||||
locator.Reload();
|
||||
return replaced;
|
||||
}
|
||||
|
||||
static DateTime Month(DateTime at) => new(at.Year, at.Month, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
static DateTime BuiltMonth(string path)
|
||||
{
|
||||
if (!File.Exists(path))
|
||||
return DateTime.MinValue;
|
||||
try
|
||||
{
|
||||
using var reader = new Reader(path, FileAccessMode.MemoryMapped);
|
||||
return Month(reader.Metadata.BuildDate);
|
||||
}
|
||||
catch (Exception ex) when (ex is InvalidDatabaseException or IOException)
|
||||
{
|
||||
return DateTime.MinValue;
|
||||
}
|
||||
}
|
||||
|
||||
// False when that month is not published (yet); throws when it is published but cannot be used.
|
||||
async Task<bool> Install(string kind, DateTime month, string path, StatisticsOptions options, CancellationToken token)
|
||||
{
|
||||
var url = $"{options.Geo.DownloadBase.TrimEnd('/')}/dbip-{kind}-lite-{month:yyyy-MM}.mmdb.gz";
|
||||
var packed = path + ".gz.part";
|
||||
var unpacked = path + ".new";
|
||||
try
|
||||
{
|
||||
using (var response = await _http.CreateClient(ClientName).GetAsync(url, HttpCompletionOption.ResponseHeadersRead, token))
|
||||
{
|
||||
if (response.StatusCode == HttpStatusCode.NotFound)
|
||||
return false;
|
||||
response.EnsureSuccessStatusCode();
|
||||
await using var source = await response.Content.ReadAsStreamAsync(token);
|
||||
await using var target = File.Create(packed);
|
||||
await CopyCapped(source, target, options.Geo.MaxDownloadBytes, token);
|
||||
}
|
||||
await using (var source = new GZipStream(File.OpenRead(packed), CompressionMode.Decompress))
|
||||
await using (var target = File.Create(unpacked))
|
||||
await CopyCapped(source, target, options.Geo.MaxDatabaseBytes, token);
|
||||
using (var reader = new Reader(unpacked, FileAccessMode.MemoryMapped))
|
||||
if (!reader.Metadata.DatabaseType.Contains(kind, StringComparison.OrdinalIgnoreCase))
|
||||
throw new InvalidDatabaseException($"{url} is a {reader.Metadata.DatabaseType} database, not {kind}");
|
||||
File.Move(unpacked, path, overwrite: true);
|
||||
return true;
|
||||
}
|
||||
finally
|
||||
{
|
||||
File.Delete(packed);
|
||||
File.Delete(unpacked);
|
||||
}
|
||||
}
|
||||
|
||||
static async Task CopyCapped(Stream source, Stream target, long limit, CancellationToken token)
|
||||
{
|
||||
var buffer = new byte[81920];
|
||||
long total = 0;
|
||||
int read;
|
||||
while ((read = await source.ReadAsync(buffer, token)) > 0)
|
||||
{
|
||||
total += read;
|
||||
if (total > limit)
|
||||
throw new InvalidDataException($"more than {limit} bytes");
|
||||
await target.WriteAsync(buffer.AsMemory(0, read), token);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
namespace PrivaPub.Infrastructure
|
||||
{
|
||||
public enum RegistrationMode
|
||||
{
|
||||
Invitations,
|
||||
Open
|
||||
}
|
||||
|
||||
// Who may make a root login. NodeInfo, both instance API versions and the sign-up endpoint read this one switch, so they
|
||||
// cannot disagree. A group invitation always creates an account, whatever the mode.
|
||||
public class RegistrationOptions
|
||||
{
|
||||
public RegistrationMode Mode { get; set; } = RegistrationMode.Invitations;//owner decision (2026-10-04): invitations only
|
||||
|
||||
public bool IsOpen => Mode == RegistrationMode.Open;
|
||||
|
||||
public const string ClosedMessage = "Sign-up is by invitation only: ask someone here for a group invitation.";
|
||||
}
|
||||
}
|
||||
@@ -6,9 +6,19 @@ namespace PrivaPub.Infrastructure.Statistics
|
||||
public string GeoDirectory { get; set; } = "/var/lib/privapub/geo";
|
||||
public int PublicCityMinUsers { get; set; } = 10;
|
||||
public CrawlerOptions Crawler { get; set; } = new();
|
||||
public GeoOptions Geo { get; set; } = new();
|
||||
}
|
||||
|
||||
//owner decision: off by default; when on, one server a minute, each at most weekly, at most MaxHosts servers
|
||||
//owner decision (2026-10-04): the server keeps its DB-IP Lite databases current by itself
|
||||
public class GeoOptions
|
||||
{
|
||||
public bool AutoUpdate { get; set; } = true;
|
||||
public string DownloadBase { get; set; } = "https://download.db-ip.com/free";
|
||||
public long MaxDownloadBytes { get; set; } = 300L * 1024 * 1024;
|
||||
public long MaxDatabaseBytes { get; set; } = 1024L * 1024 * 1024;
|
||||
}
|
||||
|
||||
//owner decision: off by default (on in production since 2026-10-04); when on, one server a minute, each at most weekly, at most MaxHosts servers
|
||||
public class CrawlerOptions
|
||||
{
|
||||
public bool Enabled { get; set; }
|
||||
|
||||
@@ -41,7 +41,8 @@ namespace PrivaPub.Middleware
|
||||
{
|
||||
return service
|
||||
.Configure<MongoSettings>(configuration.GetSection(nameof(MongoSettings)))
|
||||
.Configure<AppConfiguration>(configuration.GetSection(nameof(AppConfiguration)));
|
||||
.Configure<AppConfiguration>(configuration.GetSection(nameof(AppConfiguration)))
|
||||
.Configure<PrivaPub.Infrastructure.RegistrationOptions>(configuration.GetSection("Registrations"));
|
||||
}
|
||||
public static IServiceCollection PrivaPubWorkersConfiguration(this IServiceCollection service)
|
||||
{
|
||||
@@ -107,8 +108,16 @@ namespace PrivaPub.Middleware
|
||||
.AddSingleton<IJobHandler, DeliveryJobHandler>()
|
||||
.AddHostedService<JobWorker>();
|
||||
}
|
||||
public static IServiceCollection PrivaPubStatisticsConfiguration(this IServiceCollection service, IConfiguration configuration) =>
|
||||
service
|
||||
public static IServiceCollection PrivaPubStatisticsConfiguration(this IServiceCollection service, IConfiguration configuration)
|
||||
{
|
||||
// not federation: the DB-IP download only (GeoUpdater), plain HTTPS to a fixed host
|
||||
service.AddHttpClient(GeoUpdater.ClientName, (provider, client) =>
|
||||
{
|
||||
var baseAddress = provider.GetRequiredService<IOptionsMonitor<AppConfiguration>>().CurrentValue.BackendBaseAddress?.TrimEnd('/');
|
||||
client.Timeout = TimeSpan.FromMinutes(15);
|
||||
client.DefaultRequestHeaders.UserAgent.ParseAdd($"PrivaPub/{BuildInfo.Ref} (+{baseAddress}/)");
|
||||
});
|
||||
return service
|
||||
.Configure<StatisticsOptions>(configuration.GetSection("Statistics"))
|
||||
.AddSingleton<InteractionSalts>()
|
||||
.AddSingleton<InteractionLedger>()
|
||||
@@ -116,10 +125,12 @@ namespace PrivaPub.Middleware
|
||||
.AddHostedService(services => services.GetRequiredService<InteractionLedger>())
|
||||
.AddSingleton<IJobHandler, RollupJob>()
|
||||
.AddSingleton<IGeoLocator, DbIpLocator>()
|
||||
.AddHostedService<GeoUpdater>()
|
||||
.AddSingleton<Domain.Statistics.StatisticsQueries>()
|
||||
.AddSingleton<IJobHandler, Federation.Crawler.CrawlPlanner>()
|
||||
.AddSingleton<IJobHandler, Federation.Crawler.InstanceCrawler>()
|
||||
.AddHostedService<StatisticsSchedule>();
|
||||
}
|
||||
|
||||
public static IServiceCollection PrivaPubAuthServicesConfiguration(this IServiceCollection service, IConfiguration configuration)
|
||||
{
|
||||
|
||||
+9
-6
@@ -87,12 +87,6 @@ try
|
||||
EntityMaps.Warm();
|
||||
await DB.Default.MigrateAsync<Program>();
|
||||
await Indexes.Create();
|
||||
|
||||
if (args is ["admin", ..])
|
||||
{
|
||||
Environment.ExitCode = await AdminCommands.Run(args[1..]);
|
||||
return;
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -111,6 +105,15 @@ try
|
||||
throw;
|
||||
}
|
||||
|
||||
// Commands get every service but start nothing: no Kestrel, no hosted services, no media directory. The deploy runs
|
||||
// them as its own user, which can read the configuration and reach the private mongod but owns no www-data directory.
|
||||
if (args is ["admin", ..])
|
||||
{
|
||||
using var scope = app.Services.CreateScope();
|
||||
Environment.ExitCode = await AdminCommands.Run(args[1..], scope.ServiceProvider);
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var localizationService = app.Services.GetService<RequestLocalizationOptionsService>();
|
||||
|
||||
@@ -73,7 +73,7 @@ namespace PrivaPub.Services.ClientToServer.Private
|
||||
if (!await _localActors.TryReserveUserName(userName, LocalActorKind.Person, newAvatar.ID, default))
|
||||
return result.Invalidate(_localizer["The username '{0}' is already take.", userName]);
|
||||
if (form.Settings is { IsDefault: false })
|
||||
newAvatar.Settings = ToSettings(form.Settings);
|
||||
Apply(newAvatar.Settings, form.Settings);
|
||||
|
||||
var actor = _localActors.FromAvatar(newAvatar);
|
||||
newAvatar.Url = actor.Uri;
|
||||
@@ -116,7 +116,7 @@ namespace PrivaPub.Services.ClientToServer.Private
|
||||
avatar.Fields = form.Fields ?? new();
|
||||
avatar.PersonalNote = form.PersonalNote;
|
||||
if (form.Settings != default)
|
||||
avatar.Settings = ToSettings(form.Settings);
|
||||
Apply(avatar.Settings ??= new(), form.Settings);
|
||||
avatar.UpdatedAt = DateTime.UtcNow;
|
||||
await DB.Default.SaveAsync(avatar);
|
||||
await _outbox.PublishProfile(_localActors.FromAvatar(avatar), default);
|
||||
@@ -158,18 +158,20 @@ namespace PrivaPub.Services.ClientToServer.Private
|
||||
.Match(ru => !ru.DeletedAt.HasValue && !ru.IsBanned)
|
||||
.ExecuteAnyAsync();
|
||||
|
||||
static AvatarSettings ToSettings(ViewAvatarSettings settings) => new()
|
||||
// The client API carries only the theme; the rest (discoverable, locked, quote policy…) is set through the Mastodon API
|
||||
// and must survive an edit made here.
|
||||
static void Apply(AvatarSettings target, ViewAvatarSettings settings)
|
||||
{
|
||||
IsDefault = settings.IsDefault,
|
||||
DarkThemeIndexColour = settings.DarkThemeIndexColour,
|
||||
IconsThemeIndexColour = settings.IconsThemeIndexColour,
|
||||
LanguageCode = settings.LanguageCode,
|
||||
LightThemeIndexColour = settings.LightThemeIndexColour,
|
||||
PreferSystemTheming = settings.PreferSystemTheming,
|
||||
ThemeIsDarkGray = settings.ThemeIsDarkGray,
|
||||
ThemeIsDarkMode = settings.ThemeIsDarkMode,
|
||||
ThemeIsLightGray = settings.ThemeIsLightGray
|
||||
};
|
||||
target.IsDefault = settings.IsDefault;
|
||||
target.DarkThemeIndexColour = settings.DarkThemeIndexColour;
|
||||
target.IconsThemeIndexColour = settings.IconsThemeIndexColour;
|
||||
target.LanguageCode = settings.LanguageCode;
|
||||
target.LightThemeIndexColour = settings.LightThemeIndexColour;
|
||||
target.PreferSystemTheming = settings.PreferSystemTheming;
|
||||
target.ThemeIsDarkGray = settings.ThemeIsDarkGray;
|
||||
target.ThemeIsDarkMode = settings.ThemeIsDarkMode;
|
||||
target.ThemeIsLightGray = settings.ThemeIsLightGray;
|
||||
}
|
||||
|
||||
ViewAvatar ToView(Avatar avatar) => new()
|
||||
{
|
||||
|
||||
@@ -133,14 +133,18 @@ namespace PrivaPub.Web.Pages
|
||||
readonly Microsoft.Extensions.Options.IOptionsMonitor<Infrastructure.Statistics.StatisticsOptions> _options;
|
||||
readonly Microsoft.Extensions.Options.IOptionsMonitor<Models.AppConfiguration> _app;
|
||||
|
||||
readonly Infrastructure.Geo.IGeoLocator _geo;
|
||||
|
||||
public StargazingModel(Microsoft.Extensions.Options.IOptionsMonitor<Infrastructure.Statistics.StatisticsOptions> options,
|
||||
Microsoft.Extensions.Options.IOptionsMonitor<Models.AppConfiguration> app)
|
||||
Microsoft.Extensions.Options.IOptionsMonitor<Models.AppConfiguration> app, Infrastructure.Geo.IGeoLocator geo)
|
||||
{
|
||||
_options = options;
|
||||
_app = app;
|
||||
_geo = geo;
|
||||
}
|
||||
|
||||
public bool CrawlerEnabled => _options.CurrentValue.Crawler.Enabled;
|
||||
public string GeoSource => _geo.Source;
|
||||
public string UserAgent => Federation.Crawler.Stargazer.UserAgent(_app.CurrentValue.BackendBaseAddress);
|
||||
|
||||
public void OnGet() => Harden();
|
||||
|
||||
@@ -15,6 +15,15 @@
|
||||
<p>A server we exchange activities with is described once a week, from its public NodeInfo and, when it has one, its
|
||||
Mastodon instance API. Its location comes from the address we reached, looked up in an offline database: only the
|
||||
country is shown publicly for small servers, and only the CDN for servers behind one.</p>
|
||||
@if (Model.GeoSource is { } geo)
|
||||
{
|
||||
<p>Locations come from <a href="https://db-ip.com" rel="nofollow noopener">@geo</a>, licensed under
|
||||
<a href="https://creativecommons.org/licenses/by/4.0/" rel="nofollow noopener">CC BY 4.0</a>.</p>
|
||||
}
|
||||
else
|
||||
{
|
||||
<p>No location database is loaded yet, so servers are not located.</p>
|
||||
}
|
||||
</article>
|
||||
<article>
|
||||
<h2>The crawler</h2>
|
||||
|
||||
@@ -2,6 +2,26 @@
|
||||
"Media": {
|
||||
"Root": "/var/lib/privapub/media"
|
||||
},
|
||||
"Registrations": {
|
||||
"Mode": "Invitations"
|
||||
},
|
||||
"Statistics": {
|
||||
"Crawler": {
|
||||
"Enabled": true,
|
||||
"Seeds": [
|
||||
"mastodon.social",
|
||||
"fosstodon.org",
|
||||
"mas.to",
|
||||
"lemmy.world",
|
||||
"lemmy.ml",
|
||||
"misskey.io",
|
||||
"pixelfed.social",
|
||||
"framatube.org",
|
||||
"piefed.social",
|
||||
"bookwyrm.social"
|
||||
]
|
||||
}
|
||||
},
|
||||
"MongoSettings": {
|
||||
"Database": "PrivaPub",
|
||||
"LogsDatabase": "logs",
|
||||
|
||||
Reference in new issue
Block a user