diff --git a/CLAUDE.md b/CLAUDE.md index d866d53..3cbc629 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -414,6 +414,13 @@ tools/pasture/run.sh down # removes e (`federation: none`) until `admin/update-meta` says `all`, which `misskey_up` does. Its API is `POST /api/` with the token as `i` (`mk` in the scenario); `users/relation` answers a list, `users/notes` leaves replies out unless asked, and a user has one reaction per note. Never verify a delivery with `ap/show`: it fetches. 35 checks. +- **Lemmy (1.0.0-beta.2):** the backend alone on the shared Postgres, its admin made by `setup` in the generated + `config.hjson`. It trusts Caddy's CA through `SSL_CERT_FILE` and reaches the network through + `DANGER_FEDERATION_ALLOW_LOCAL_IP=1`; 0.19 cannot join (its rustls trusts only its bundled roots). Its API is + `/api/v4/` with a bearer token (`lm` in the scenario), and `sort` values are lowercase. It logs no refused + activity at `warn` (`LEMMY_LOG` sets `RUST_LOG`); the reason is in the 400's body. It answers our community's echo of + its own activity and every bare `Announce{object}` 400 by design, and the echo is still needed (see + `docs/INTEROP.md`, Lemmy). 20 checks; relayed votes and a moderator's removal are expected failures (P7). - **Crawler:** `PRIVAPUB_ENV="Statistics__Crawler__Enabled=true Statistics__Crawler__Seeds__0=mastodon.test" run.sh up mastodon`, then `interop.sh crawler`. `PRIVAPUB_ENV` passes any setting to the PrivaPub container. - `run.sh up` replaces every container, Mongo included, so each run starts clean. To keep the data, republish into diff --git a/PrivaPub/Federation/Outbox/GroupDistributor.cs b/PrivaPub/Federation/Outbox/GroupDistributor.cs index 01dd7c6..caf9f76 100644 --- a/PrivaPub/Federation/Outbox/GroupDistributor.cs +++ b/PrivaPub/Federation/Outbox/GroupDistributor.cs @@ -40,6 +40,8 @@ namespace PrivaPub.Federation.Outbox ["audience"] = group.Uri, ["object"] = embedded }; + // Every follower, the author's own server included: Lemmy 1.0 keeps its user's post pending until the community + // announces it back, and clears that before it answers the echo 400 ("Object is not remote"). await _delivery.EnqueueToFollowers(group, announce, token); if (isNewPost && !string.IsNullOrEmpty(objectUri)) await _delivery.EnqueueToFollowers(group, ActivityPubRenderer.Announce(group, objectUri, $"announce-object-{key}"), token); diff --git a/docs/INTEROP.md b/docs/INTEROP.md index 3e2c37f..68b3fab 100644 --- a/docs/INTEROP.md +++ b/docs/INTEROP.md @@ -427,7 +427,7 @@ on a Page: pins live in `featured`, locks in `Lock`. | Link posts: keep `Link.href`, the thumbnail `image` and alt text; build the card (Lemmy sends no title or description for the link) | P1 | `Status.card` | | `ChatMessage` in and out (out only to Lemmy < 1.0 and Mbin; `Note` to everyone else) | P1 | `visibility: direct` | | Outbound shape for Lemmy: both `to` and `cc`; the community in `to`; Public in the object, Create and Announce; votes and comments sent to the community inbox | P1 | — | -| Communities we host: announce to every follower instance including the author's; pick `Announce(object)` per peer by NodeInfo (as PieFed does) | P1 | — | +| Communities we host: pick `Announce(object)` per peer by NodeInfo (as PieFed does). Announcing to every follower instance, the author's included, is done and needed (pasture evidence below) | P1 | — | | Flags from a `Service`-typed reporter actor with `to: [community]`; the reporter stays anonymous | P2 | — | | `Warn` → `moderation_warning` notification; `Resolve{Flag}` | P2 | AccountWarning | | Remote communities: `description`, `language[]`, private (`locked`), `discoverable`; post tags | P2 | `Account.locked`, own `privapub.flairs[]` | @@ -435,6 +435,31 @@ on a Page: pins live in `featured`, locks in `Lock`. | `Feed` actors | P2 | group-like account | | Read 1.0 `context`, grouped by root post; cross-post detection by URL | P3 | — | +**Pasture evidence (2026-10-03, Lemmy 1.0.0-beta.2, `tools/pasture/scenarios/lemmy.sh`):** 20 checks pass and 3 are +expected failures: +- communities both ways: Lemmy follows ours and alice follows Lemmy's, each Accept arriving; +- a Lemmy thread in our community arrives with its title, and our titled community post reaches Lemmy; +- the Lemmy community's Announce brings its thread to alice's home; +- alice's post mentioning a Lemmy community lands in it, titled from its first line (Lemmy repeats that line in the + body); +- comments both ways, and alice's like counted as an upvote; +- private messages both ways: 1.0 takes our single-recipient direct `Note`, and sends its own as `Note`s; +- statistics. + +What it showed: +- **1.0 keeps its user's thread in a remote community `federation_pending` until the community announces it back**, and + clears the flag *before* answering that echo 400 (`Object is not remote`). Without the echo the thread stays pending, + so `GroupDistributor` sends the `Announce{Create}` to the author's own server on purpose. +- **Every bare `Announce{object}` is answered 400** (`Failed to parse object`: Lemmy dereferences it expecting an + activity), as Lemmy answers the compatibility `Announce(Page)` it sends itself. Both 400s show up as dead deliveries + in the statistics. +- **Votes travel only to the community**, which relays them as `Announce{Like}` and `Announce{Dislike}`. They are + dropped as `unsupported` until P7 (expected failures), as is a moderator's removal. +- Lemmy logs no refused activity at `warn`; the reason is in the 400's body, which our delivery does not keep. The + scenario's API notes: `sort` values are lowercase (`new`), private messages and mentions are in + `account/notification/list`, and `resolve_object` takes both `!community@host` and `@user@host`. +- 0.19 cannot join the pasture: its rustls trusts only its bundled roots, never Caddy's CA. + ### PieFed 1.7.17 and Mbin 1.10.1 - **PieFed sends Lemmy's set plus:** diff --git a/tools/pasture/interop.sh b/tools/pasture/interop.sh index 20a269a..49d9245 100755 --- a/tools/pasture/interop.sh +++ b/tools/pasture/interop.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash # Drives PrivaPub and each peer (started by run.sh) through each other's federation, as their users would, then checks # that PrivaPub's statistics saw it all without naming anyone. -# usage: tools/pasture/interop.sh [peer...] peers: gts (default) +# usage: tools/pasture/interop.sh [peer...] peers: gts (default), mastodon, misskey, lemmy, crawler set -uo pipefail here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" . "$here/lib/interop.sh" diff --git a/tools/pasture/peers/lemmy.sh b/tools/pasture/peers/lemmy.sh new file mode 100644 index 0000000..8662200 --- /dev/null +++ b/tools/pasture/peers/lemmy.sh @@ -0,0 +1,36 @@ +# Lemmy 1.0 (beta): the backend alone (no UI, no pict-rs) on the shared Postgres. 0.19's rustls ignores every CA but its +# bundled ones, so only 1.0 can trust Caddy's; DANGER_FEDERATION_ALLOW_LOCAL_IP lets it reach the pasture's addresses. +LEMMY_IMAGE=${LEMMY_IMAGE:-docker.io/dessalines/lemmy:1.0.0-beta.2} +. "$here/peers/shared.sh" + +lemmy_up() { + shared_postgres_up + podman exec pasture-postgres sh -c "psql -U pasture -tc \"select 1 from pg_database where datname='lemmy'\" | grep -q 1 || createdb -U pasture lemmy" + mkdir -p "$here/.state/lemmy" + cat > "$here/.state/lemmy/config.hjson" </dev/null + for _ in $(seq 1 90); do + site lemmy.test -s -o /dev/null -w '%{http_code}' https://lemmy.test:6443/api/v4/site 2>/dev/null | grep -q 200 && break + sleep 2 + done + site lemmy.test -s -X POST https://lemmy.test:6443/api/v4/account/auth/login -H 'Content-Type: application/json' \ + -d '{"username_or_email":"lemmyuser","password":"Lemmy-Pasture-Pass-1"}' \ + | python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])" > "$here/.state/lemmy.token" 2>/dev/null || true + echo "lemmy: https://lemmy.test:6443" +} diff --git a/tools/pasture/run.sh b/tools/pasture/run.sh index df00a35..80d0c08 100755 --- a/tools/pasture/run.sh +++ b/tools/pasture/run.sh @@ -2,7 +2,7 @@ # A private test fediverse on one podman network: PrivaPub (privapub.test) and the peers asked for, behind one Caddy # whose internal CA every side is told to accept (its root is copied to .ca/root.crt). From the workstation: PrivaPub's # API at http://127.0.0.1:6971, every site at https://.test:6443 (curl --resolve .test:6443:127.0.0.1 -k). -# usage: tools/pasture/run.sh up [peer...] | down | logs | ps peers: gts (default) +# usage: tools/pasture/run.sh up [peer...] | down | logs | ps peers: gts (default), mastodon, misskey, lemmy set -euo pipefail . "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/pasture.sh" diff --git a/tools/pasture/scenarios/lemmy.sh b/tools/pasture/scenarios/lemmy.sh new file mode 100644 index 0000000..aae0900 --- /dev/null +++ b/tools/pasture/scenarios/lemmy.sh @@ -0,0 +1,109 @@ +# Lemmy 1.0: communities both ways (FEP-1b12), threads with titles, comments both ways, votes up and down both ways, +# private messages both ways, a moderator's removal (P7), statistics. Lemmy is driven through its v4 API. +LM=https://lemmy.test:6443 +LT=$(cat "$here/.state/lemmy.token" 2>/dev/null) +# lm [json]: a Lemmy v4 API call as lemmyuser +lm() { + local method=$1 path=$2 body=${3:-} + if [ -n "$body" ]; then + site lemmy.test -s -X "$method" "$LM/api/v4/$path" -H "Authorization: Bearer $LT" -H 'Content-Type: application/json' -d "$body" + else + site lemmy.test -s -X "$method" "$LM/api/v4/$path" -H "Authorization: Bearer $LT" + fi +} +# the posts Lemmy holds in a community, and the one whose ap_id is given +lm_posts() { lm GET "post/list?community_id=$1&sort=new&limit=50" | j "print(json.dumps([p['post'] for p in (d.get('items') or d.get('posts') or [])]))"; } +lm_post_by_ap() { lm_posts "$1" | j "print(json.dumps(next((p for p in d if p['ap_id']=='$2'), None)))"; } + +echo "lemmy" +[ -n "$LT" ] && ok "Lemmy token for lemmyuser" || { ko "Lemmy token"; return 1; } +LAT=$(privapub_token alice_lemmy) +LAH="Authorization: Bearer $LAT" +[ -n "$LAT" ] && ok "PrivaPub token for alice_lemmy" || { ko "PrivaPub token for alice_lemmy"; return 1; } +jwt=$(privapub_root) +alice_id=$(curl -s -H "$LAH" "$P/api/v1/accounts/verify_credentials" | j "print(d['id'])") +dogs="dogs$(date +%s)" +cats="cats$(date +%s)" + +echo " communities" +dogs_gid=$(curl -s -X POST $P/clientapi/group/insert -H 'Content-Type: application/json' -H "Authorization: Bearer $jwt" \ + -d "{\"avatarId\":\"$alice_id\",\"userName\":\"$dogs\",\"name\":\"Pasture dogs\",\"description\":\"a PrivaPub community\",\"isCommunity\":true}" | j "print(d['id'])") +dogs_on_lm=$(lm GET "resolve_object?q=!$dogs@privapub.test" | j "print(d['community']['id'])") +[ -n "$dogs_on_lm" ] && ok "Lemmy resolves a PrivaPub community" || ko "Lemmy cannot resolve the PrivaPub community" +lm POST community/follow "{\"community_id\":$dogs_on_lm,\"follow\":true}" >/dev/null +until_true 30 '[ "$(lm GET "community?id=$dogs_on_lm" | j "print(((d[\"community_view\"].get(\"community_actions\") or {}).get(\"follow_state\") or \"\").lower())")" = "accepted" ]' \ + && ok "lemmyuser follows the PrivaPub community (Accept arrived)" || ko "Lemmy's community follow not accepted" +cats_on_lm=$(lm POST community "{\"name\":\"$cats\",\"title\":\"Pasture cats\"}" | j "print(d['community_view']['community']['id'])") +cats_on_p=$(curl -s -H "$LAH" "$P/api/v2/search?q=@$cats@lemmy.test&resolve=true&type=accounts" | j "print(d['accounts'][0]['id'])") +[ -n "$cats_on_p" ] && ok "PrivaPub resolves a Lemmy community" || ko "PrivaPub cannot resolve the Lemmy community" +curl -s -o /dev/null -X POST -H "$LAH" "$P/api/v1/accounts/$cats_on_p/follow" +until_true 30 '[ "$(curl -s -H "$LAH" "$P/api/v1/accounts/relationships?id[]=$cats_on_p" | j "print(d[0][\"following\"])")" = "True" ]' && ok "alice follows the Lemmy community (Accept arrived)" || ko "PrivaPub's community follow not accepted" + +echo " threads" +lm_thread=$(lm POST post "{\"name\":\"A Lemmy thread\",\"body\":\"posted from Lemmy into PrivaPub\",\"community_id\":$dogs_on_lm}" | j "print(d['post_view']['post']['ap_id'])") +dogs_on_p=$(curl -s -H "$LAH" "$P/api/v1/accounts/lookup?acct=$dogs" | j "print(d['id'])") +until_true 30 '[ "$(curl -s -H "$LAH" "$P/api/v1/accounts/$dogs_on_p/statuses" | j "print(any(s[\"uri\"]==\"$lm_thread\" and (s.get(\"privapub\") or {}).get(\"title\")==\"A Lemmy thread\" for s in d))")" = "True" ]' \ + && ok "a Lemmy thread in the PrivaPub community arrives with its title" || ko "Lemmy's thread missing or untitled on PrivaPub" +curl -s -o /dev/null -X POST $P/clientapi/post/insert -H 'Content-Type: application/json' -H "Authorization: Bearer $jwt" \ + -d "{\"avatarId\":\"$alice_id\",\"title\":\"A PrivaPub community thread\",\"text\":\"posted into our own community\",\"groupId\":\"$dogs_gid\"}" +dogs_post=$(podman exec pasture-mongo mongosh --quiet PrivaPub --eval 'print(db.Post.findOne({Text:/posted into our own community/}, {}, {sort:{_id:-1}}).ObjectURI)') +until_true 30 '[ "$(lm_post_by_ap "$dogs_on_lm" "$dogs_post" | j "print(d and d[\"name\"])")" = "A PrivaPub community thread" ]' \ + && ok "a titled post in the PrivaPub community reaches Lemmy through its announce" || ko "PrivaPub community post missing on Lemmy" +# Lemmy 1.0 holds its user's thread in a remote community as pending until the community announces it back, and clears +# the flag even though it then answers that Announce 400 ("Object is not remote"): the echo is what publishes it. +until_true 30 '[ "$(lm_post_by_ap "$dogs_on_lm" "$lm_thread" | j "print(d and d[\"federation_pending\"])")" = "False" ]' \ + && ok "the PrivaPub community's announce back publishes Lemmy's thread there" || ko "Lemmy's thread stays federation_pending" +lm_cats_thread=$(lm POST post "{\"name\":\"Cats only\",\"body\":\"a Lemmy community post\",\"community_id\":$cats_on_lm}" | j "print(d['post_view']['post']['ap_id'])") +until_true 30 'curl -s -H "$LAH" "$P/api/v1/timelines/home" | j "print(any(s[\"uri\"]==\"$lm_cats_thread\" or (s.get(\"reblog\") or {}).get(\"uri\")==\"$lm_cats_thread\" for s in d))" | grep -q True' \ + && ok "the Lemmy community's announce brings its thread to alice's home" || ko "Lemmy community thread missing from alice's home" +p_thread=$(curl -s -X POST -H "$LAH" $P/api/v1/statuses --data-urlencode "status=A PrivaPub thread +@$cats@lemmy.test posted from PrivaPub" -d 'visibility=public') +p_thread_id=$(echo "$p_thread" | j "print(d['id'])"); p_thread_uri=$(echo "$p_thread" | j "print(d['uri'])") +until_true 30 '[ "$(lm_post_by_ap "$cats_on_lm" "$p_thread_uri")" != "null" ]' && ok "alice's thread lands in the Lemmy community" || ko "PrivaPub thread missing from the Lemmy community" +p_thread_on_lm=$(lm_post_by_ap "$cats_on_lm" "$p_thread_uri" | j "print(d['id'])") + +echo " comments" +lm POST comment "{\"post_id\":$p_thread_on_lm,\"content\":\"a Lemmy comment on PrivaPub\"}" >/dev/null +until_true 30 '[ "$(curl -s -H "$LAH" "$P/api/v1/statuses/$p_thread_id/context" | j "print(any(\"a Lemmy comment on PrivaPub\" in s[\"content\"] for s in d[\"descendants\"]))")" = "True" ]' \ + && ok "Lemmy's comment threads under alice's post" || ko "Lemmy's comment missing on PrivaPub" +lm_cats_on_p=$(curl -s -H "$LAH" "$P/api/v1/timelines/home" | j "print(next((s.get('reblog') or s)['id'] for s in d if (s.get('reblog') or s)['uri']=='$lm_cats_thread'))") +curl -s -o /dev/null -X POST -H "$LAH" $P/api/v1/statuses -d "status=a PrivaPub comment on Lemmy&in_reply_to_id=$lm_cats_on_p&visibility=public" +lm_cats_id=$(lm GET "resolve_object?q=$lm_cats_thread" | j "print(d['post']['id'])") +until_true 30 '[ "$(lm GET "comment/list?post_id=$lm_cats_id" | j "print(any(\"a PrivaPub comment on Lemmy\" in c[\"comment\"][\"content\"] for c in (d.get(\"items\") or d.get(\"comments\") or [])))")" = "True" ]' \ + && ok "alice's reply becomes a comment on Lemmy" || ko "PrivaPub reply missing as a Lemmy comment" + +echo " votes" +# Lemmy sends a vote only to the community, which relays it inside an Announce: P7 trusts those for the community's own posts. +lm POST post/like "{\"post_id\":$p_thread_on_lm,\"is_upvote\":true}" >/dev/null +if until_true 15 '[ "$(curl -s -H "$LAH" "$P/api/v1/statuses/$p_thread_id" | j "print(d[\"favourites_count\"])")" = "1" ]'; then + ok "Lemmy's upvote counts as a like on PrivaPub" +else + xf "Lemmy's upvote counts as a like on PrivaPub (votes relayed by a community are P7)" +fi +lm POST post/like "{\"post_id\":$p_thread_on_lm,\"is_upvote\":false}" >/dev/null +if until_true 15 '[ "$(curl -s -H "$LAH" "$P/api/v1/statuses/$p_thread_id" | j "print(d[\"favourites_count\"], ((d.get(\"privapub\") or {}).get(\"votes\") or {}).get(\"down\"))")" = "0 1" ]'; then + ok "Lemmy's change to a downvote reaches PrivaPub" +else + xf "Lemmy's change to a downvote reaches PrivaPub (votes relayed by a community are P7)" +fi +curl -s -o /dev/null -X POST -H "$LAH" "$P/api/v1/statuses/$lm_cats_on_p/favourite" +until_true 30 '[ "$(lm GET "post?id=$lm_cats_id" | j "print(d[\"post_view\"][\"post\"].get(\"upvotes\"))")" = "2" ]' && ok "alice's like is an upvote on Lemmy" || ko "like not counted as an upvote on Lemmy" + +echo " private messages" +alice_on_lm=$(lm GET "resolve_object?q=@alice_lemmy@privapub.test" | j "print(d['person']['id'])") +lm POST private_message "{\"content\":\"a secret from Lemmy\",\"recipient_id\":$alice_on_lm}" >/dev/null +until_true 30 'curl -s -H "$LAH" "$P/api/v1/conversations" | grep -q "a secret from Lemmy"' && ok "Lemmy's private message arrives as a DM" || ko "Lemmy's private message missing on PrivaPub" +curl -s -o /dev/null -X POST -H "$LAH" $P/api/v1/statuses -d 'status=@lemmyuser@lemmy.test a secret from PrivaPub&visibility=direct' +until_true 30 'lm GET "account/notification/list?limit=50" | grep -q "a secret from PrivaPub"' && ok "alice's DM arrives as a Lemmy private message" || ko "DM missing on Lemmy" + +echo " moderation" +lm POST post/remove "{\"post_id\":$lm_cats_id,\"removed\":true,\"reason\":\"pasture\"}" >/dev/null +sleep 10 +if curl -s -H "$LAH" "$P/api/v1/timelines/home" | j "print(any((s.get('reblog') or s)['uri']=='$lm_cats_thread' for s in d))" | grep -q False; then + ok "a moderator's removal reaches PrivaPub" +else + xf "a moderator's removal reaches PrivaPub (the Lemmy moderation set is P7)" +fi + +echo " statistics" +stats_check lemmy.test lemmy