thepra/SocialPub
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Audio and video are processed off the request

Browse Source 
Every audio and video upload was probed and remuxed inside the request, whatever its length; ffmpeg would read any
protocol and probe any format; `-map 0` kept the data tracks iPhones add, which mp4 refuses; nothing was ever
transcoded, so HEVC or MPEG-4 Part 2 reached browsers that can't play them, and the advertised video_matrix_limit
and frame rate limit were never applied; the output was read whole into memory, the video was saved before its
poster could fail, and the poster's frame leaked in /tmp. FLAC uploads were served as 404.

Now an upload sent to /api/v2/media is stored as sent in media-incoming (beside the media root, never served) and
answered with 202 and no url, while a ProcessMedia job, one at a time, does the work; GET /api/v1/media/:id answers 206
until it is ready, or 422 with why, and media still processing can't be posted. v1 processes before answering.
ffmpeg reads only that file (protocol whitelist, format forced from the probe) and drops data and subtitle tracks. A
video browsers play as it is (H.264, VP8, VP9, AV1 within Media:MaxVideoPixels and MaxFrameRate) is remuxed, anything
else transcoded to H.264 that fits, as Mastodon does; longer than Media:MaxSeconds is refused. Outputs move into place
only once everything succeeded, every temporary file goes, durations are kept, FLAC is served as audio/flac, and the
unit gets PrivateTmp. The instance API advertises the limits that are now applied.

No pasture scenario uploads audio or video through PrivaPub, so the sweep could not see this. MastodonMediaTests: v2
answers 202 then the job makes it playable (and an unreadable file 422 once processed), media still processing can't
be posted, MPEG-4 Part 2 becomes H.264, a video over the limit is made smaller, FLAC is served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
This commit is contained in:
thepraandClaude Opus 5.5 committed 2026-10-07 10:48:56 +02:00
1 parent e4f9d0b61e
commit 8e59145826
14 files changed
+281 -48
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

PrivaPub/Domain/Media/MediaService.cs
+123 -28
View File
@@ -38,7 +38,12 @@ namespace PrivaPub.Domain.Media
/// outside what /media/files serves. (Not a dot-prefixed folder inside Root: the file provider only hides a file whose
/// own name starts with a dot.)</summary>
string TrashRoot { get; }
Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, CancellationToken token);
/// <summary>Audio and video waiting for their processing: beside the media root, never served.</summary>
string IncomingRoot { get; }
/// <summary>An upload; with <paramref name="later"/>, audio and video are only stored, "pending" for ProcessMedia.</summary>
Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, bool later, CancellationToken token);
/// <summary>Processes a pending upload's audio or video; false when it can't be (the row then says why).</summary>
Task<bool> ProcessPending(MediaAttachment pending, CancellationToken token);
/// <summary>A persona's new avatar or header ("avatar" or "header"), cropped to its size, as a row of its own.</summary>
Task<MediaOutcome> ProfileImage(string avatarId, string kind, IFormFile file, int width, int height, CancellationToken token);
/// <summary>Trashes the media <paramref name="which"/> matches (and that isn't trashed yet): each row is marked in one
@@ -64,6 +69,12 @@ namespace PrivaPub.Domain.Media
["audio/mp4"] = "m4a", ["audio/x-m4a"] = "m4a", ["audio/flac"] = "flac", ["audio/webm"] = "webm"
};
// what /media/files says each file is: ASP.NET's map, which has no entry for FLAC
public static Microsoft.AspNetCore.StaticFiles.FileExtensionContentTypeProvider ContentTypes { get; } = new()
{
Mappings = { [".flac"] = "audio/flac" }
};
// what an upload may be, as the instance API advertises it
public static IReadOnlyList<string> SupportedTypes => ImageTypes.Concat(AvTypes.Keys).ToList();
@@ -97,9 +108,13 @@ namespace PrivaPub.Domain.Media
public string TrashRoot => Root.TrimEnd(Path.DirectorySeparatorChar) + "-trash";
public string IncomingRoot => Root.TrimEnd(Path.DirectorySeparatorChar) + "-incoming";
string Incoming(string id) => Path.Combine(IncomingRoot, id + ".in");
public string Url(string relativePath) => relativePath == default ? default : $"{_localActors.BaseAddress}/media/files/{relativePath.Replace('\\', '/')}";
public async Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, CancellationToken token)
public async Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, bool later, CancellationToken token)
{
if (file == default || file.Length == 0)
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File can't be blank");
@@ -152,13 +167,25 @@ namespace PrivaPub.Domain.Media
_processing.Release();
}
}
else if (contentType != default && AvTypes.TryGetValue(contentType, out var extension))
else if (contentType != default && AvTypes.ContainsKey(contentType))
{
if (file.Length > options.MaxVideoBytes)
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File is too big");
var outcome = await ProcessAv(file, extension, contentType, attachment, token);
if (!outcome.Ok)
return outcome;
// stored as it came, outside what is served, then processed: by a job when asked later (v2), here otherwise (v1)
attachment.ID = (string)attachment.GenerateNewID();
attachment.Kind = contentType.StartsWith("video/", StringComparison.Ordinal) ? "video" : "audio";
attachment.ContentType = contentType;
Directory.CreateDirectory(IncomingRoot);
await using (var stored = File.Create(Incoming(attachment.ID)))
await file.CopyToAsync(stored, token);
if (later)
attachment.ProcessingState = "pending";
else
{
var outcome = await ProcessIncoming(attachment, token);
if (!outcome.Ok)
return outcome;
}
}
else
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File type is not supported");
@@ -251,6 +278,8 @@ namespace PrivaPub.Domain.Media
foreach (var full in new[] { Path.Combine(TrashRoot, relative), Path.Combine(Root, relative) })
if (File.Exists(full))
File.Delete(full);
if (File.Exists(Incoming(trashed.ID)))
File.Delete(Incoming(trashed.ID));
await DB.Default.DeleteAsync<MediaAttachment>(m => m.ID == trashed.ID && m.TrashedAt != null);
}
@@ -362,37 +391,102 @@ namespace PrivaPub.Domain.Media
return bands.Format == Enums.BandFormat.Uchar ? bands.Copy() : bands.Cast(Enums.BandFormat.Uchar);
}
async Task<MediaOutcome> ProcessAv(IFormFile file, string extension, string contentType, MediaAttachment attachment, CancellationToken token)
public async Task<bool> ProcessPending(MediaAttachment pending, CancellationToken token)
{
var outcome = await ProcessIncoming(pending, token);
if (outcome.Ok)
{
await DB.Default.Update<MediaAttachment>().Match(m => m.ID == pending.ID && m.TrashedAt == null)
.Modify(m => m.Kind, pending.Kind)
.Modify(m => m.ContentType, pending.ContentType)
.Modify(m => m.FilePath, pending.FilePath)
.Modify(m => m.PreviewPath, pending.PreviewPath)
.Modify(m => m.Size, pending.Size)
.Modify(m => m.Width, pending.Width)
.Modify(m => m.Height, pending.Height)
.Modify(m => m.Blurhash, pending.Blurhash)
.Modify(m => m.DurationSeconds, pending.DurationSeconds)
.Modify(m => m.ProcessingState, null)
.ExecuteAsync(token);
return true;
}
await DB.Default.Update<MediaAttachment>().MatchID(pending.ID)
.Modify(m => m.ProcessingState, "failed").Modify(m => m.ProcessingError, outcome.Error).ExecuteAsync(token);
return false;
}
// the stored upload, made playable: probed, then remuxed without its metadata when browsers play it as it is (H.264,
// VP8, VP9 or AV1 within MaxVideoPixels and MaxFrameRate), or else transcoded to H.264 that fits; a frame becomes the
// poster. ffmpeg only ever reads the local file (no protocol but file, its format forced from the probe). Nothing is
// saved until everything succeeded, and every temporary file goes, the stored upload too.
async Task<MediaOutcome> ProcessIncoming(MediaAttachment attachment, CancellationToken token)
{
var options = _options.CurrentValue;
var input = Incoming(attachment.ID);
var temp = Path.Combine(Path.GetTempPath(), $"privapub-{Guid.NewGuid():N}");
var input = temp + ".in";
var output = temp + "." + extension;
var frame = temp + ".png";
var output = default(string);
await _processing.WaitAsync(token);
try
{
await using (var target = File.Create(input))
await file.CopyToAsync(target, token);
var probe = await FFProbe.AnalyseAsync(input, cancellationToken: token);
var isVideo = contentType.StartsWith("video/") && probe.PrimaryVideoStream != default;
await FFMpegArguments.FromFileInput(input)
.OutputToFile(output, true, o => o.WithCustomArgument("-map 0 -map_metadata -1 -map_chapters -1 -c copy" + (extension is "mp4" or "m4a" ? " -movflags +faststart" : string.Empty)))
var probe = await FFProbe.AnalyseAsync(input, default, token, "-protocol_whitelist file");
if (probe.Duration > TimeSpan.FromSeconds(options.MaxSeconds))
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file is too long");
var video = attachment.ContentType.StartsWith("video/", StringComparison.Ordinal) ? probe.PrimaryVideoStream : default;
var audio = probe.PrimaryAudioStream;
if (video == default && audio == default)
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file could not be processed");
var extension = AvTypes[attachment.ContentType];
var format = probe.Format.FormatName.Split(',')[0];
string arguments;
if (video == default)
arguments = "-map 0:a:0 -vn -dn -sn -map_metadata -1 -map_chapters -1 -c copy" + (extension == "m4a" ? " -movflags +faststart" : string.Empty);
else if (video.CodecName is "h264" or "vp8" or "vp9" or "av1" && (long)video.Width * video.Height <= options.MaxVideoPixels
&& video.FrameRate <= options.MaxFrameRate + 0.5)
arguments = "-map 0:v:0 -map 0:a:0? -dn -sn -map_metadata -1 -map_chapters -1 -c copy" + (extension == "mp4" ? " -movflags +faststart" : string.Empty);
else
{
var scale = Math.Min(1, Math.Sqrt(options.MaxVideoPixels / (double)((long)video.Width * video.Height)));
var width = Math.Max(2, (int)(video.Width * scale) / 2 * 2);
var height = Math.Max(2, (int)(video.Height * scale) / 2 * 2);
extension = "mp4";
arguments = $"-map 0:v:0 -map 0:a:0? -dn -sn -map_metadata -1 -map_chapters -1 -c:v libx264 -preset veryfast -crf 23 -pix_fmt yuv420p "
+ $"-vf scale={width}:{height} -fpsmax {options.MaxFrameRate.ToString(CultureInfo.InvariantCulture)} -c:a aac -b:a 128k -movflags +faststart";
}
output = temp + "." + extension;
await FFMpegArguments.FromFileInput(input, true, o => o.ForceFormat(format).WithCustomArgument("-protocol_whitelist file"))
.OutputToFile(output, true, o => o.WithCustomArgument(arguments))
.CancellableThrough(token)
.ProcessAsynchronously();
attachment.Kind = isVideo ? "video" : "audio";
attachment.ContentType = extension switch { "mp4" => "video/mp4", "webm" => isVideo ? "video/webm" : "audio/webm", "m4a" => "audio/mp4", "mp3" => "audio/mpeg", _ => contentType };
attachment.FilePath = await Save(await File.ReadAllBytesAsync(output, token), extension, token);
attachment.Size = new FileInfo(output).Length;
if (isVideo)
var made = await FFProbe.AnalyseAsync(output, default, token, "-protocol_whitelist file");
byte[] poster = default;
if (video != default)
{
attachment.Width = probe.PrimaryVideoStream.Width;
attachment.Height = probe.PrimaryVideoStream.Height;
var frame = temp + ".png";
await FFMpeg.SnapshotAsync(input, frame, captureTime: TimeSpan.FromSeconds(Math.Min(1, probe.Duration.TotalSeconds / 2)));
var still = ProcessImage(await File.ReadAllBytesAsync(frame, token), 640, _options.CurrentValue.PreviewSide);
attachment.PreviewPath = await Save(still.Preview, "jpg", token);
attachment.Width = made.PrimaryVideoStream?.Width;
attachment.Height = made.PrimaryVideoStream?.Height;
await FFMpeg.SnapshotAsync(output, frame, captureTime: TimeSpan.FromSeconds(Math.Min(1, made.Duration.TotalSeconds / 2)));
var still = ProcessImage(await File.ReadAllBytesAsync(frame, token), 640, options.PreviewSide);
poster = still.Preview;
attachment.Blurhash = still.Blurhash;
File.Delete(frame);
}
attachment.Kind = video != default ? "video" : "audio";
attachment.ContentType = extension switch
{
"mp4" => "video/mp4",
"webm" => video != default ? "video/webm" : "audio/webm",
"m4a" => "audio/mp4",
"mp3" => "audio/mpeg",
"ogg" => "audio/ogg",
"wav" => "audio/wav",
"flac" => "audio/flac",
_ => attachment.ContentType
};
attachment.DurationSeconds = Math.Round(made.Duration.TotalSeconds, 2);
attachment.Size = new FileInfo(output).Length;
attachment.FilePath = SaveFile(output, extension);
if (poster != default)
attachment.PreviewPath = await Save(poster, "jpg", token);
return new MediaOutcome(attachment);
}
catch (Exception ex) when (ex is not OperationCanceledException)
@@ -402,7 +496,8 @@ namespace PrivaPub.Domain.Media
}
finally
{
foreach (var path in new[] { input, output })
_processing.Release();
foreach (var path in new[] { input, output, frame }.Where(p => p != default))
if (File.Exists(path))
File.Delete(path);
}