Add audio translation task type and provider by julien-nc · Pull Request #335 · nextcloud/integration_openai

julien-nc · 2026-02-04T16:11:50Z

New audio2audio:translate task type (should be created in server?)
Audio translation provider
Factorize translation logic in a service
Use correct user language for IL10N text translations happening in the task

kyteinsky · 2026-02-04T18:08:05Z

+				$this->logger->warning('Text to speech generation failed: no speech returned');
+				throw new ProcessingException('Text to speech generation failed: no speech returned');
+			}
+			$translatedAudio = $includeWatermark ? $this->watermarkingService->markAudio($apiResponse['body']) : $apiResponse['body'];


maybe better to watermark the transcript of the input audio so the translated text and the translated audio both have the watermark in the target language

The audio should always be marked directly as there is specific metadata that is added to the audio.

kyteinsky · 2026-02-04T18:13:18Z

+		if ($includeWatermark) {
+			if ($userId !== null) {
+				$user = $this->userManager->getExistingUser($userId);
+				$lang = $this->l10nFactory->getUserLanguage($user);
+				$l = $this->l10nFactory->get(Application::APP_ID, $lang);
+				$ttsPrompt .= "\n\n" . $l->t('This was generated using Artificial Intelligence.');
+			} else {
+				$ttsPrompt .= "\n\n" . $this->l->t('This was generated using Artificial Intelligence.');
+			}
+		}


this can also work but it would add the text/audio in the user's language which may or may not be the target language.

The watermark is now appended to the input transcript in the language of the user. So it ends up being in the audio and text outputs in the target language.

julien-nc · 2026-06-15T10:21:24Z

I had to update the composer dependencies to fix CI issues.

lukasdotcom

Didn't look that deeply yet, but found a few things. Also it might be worth it to use the new methods in TranslateService for the old task processing provider too.

lukasdotcom · 2026-06-15T13:53:43Z

+				$this->logger->warning('Text to speech generation failed: no speech returned');
+				throw new ProcessingException('Text to speech generation failed: no speech returned');
+			}
+			$translatedAudio = $includeWatermark ? $this->watermarkingService->markAudio($apiResponse['body']) : $apiResponse['body'];


The audio should always be marked directly as there is specific metadata that is added to the audio.

marcelklehr · 2026-06-15T14:45:40Z

A test for the AudioToAudioTranslate provider would be nice, but I'm fine if we go without :)

marcelklehr · 2026-06-15T14:46:29Z

+use OCP\TaskProcessing\ShapeDescriptor;
+
+class AudioToAudioTranslateTaskType implements ITaskType {
+	public const ID = Application::APP_ID . ':audio2audio:translate';


Any reason to have this not in server?

(Except that server PRs are annoying :D )

I'll add it later and adjust here.

…, factorize translation logic in the translate service Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

…rmarked in the target language Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

use TranslateService::getStaticLanguages() to get the input shape enum values watermark the audio directly keep the watermark suffix in the language of the user who scheduled register audio translation provider if tts is enabled fix reporting translation output fix cache being created inside a loop cleanup Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

julien-nc requested review from kyteinsky and marcelklehr February 4, 2026 16:11

julien-nc added enhancement New feature or request 3. to review labels Feb 4, 2026

julien-nc changed the title ~~Add audio translation task type and provider…~~ Add audio translation task type and provider Feb 4, 2026

julien-nc force-pushed the enh/noid/audio-translation branch 4 times, most recently from 0c8d7be to 2e45e17 Compare February 4, 2026 16:40

kyteinsky reviewed Feb 4, 2026

View reviewed changes

kyteinsky mentioned this pull request Feb 6, 2026

Translation provider prompt should be engineered a bit more to avoid outputting explanations #336

Open

julien-nc marked this pull request as draft February 24, 2026 09:41

julien-nc force-pushed the enh/noid/audio-translation branch 3 times, most recently from 67bfb51 to 1849a43 Compare June 15, 2026 09:41

julien-nc requested a review from kyteinsky June 15, 2026 10:19

julien-nc marked this pull request as ready for review June 15, 2026 10:19

marcelklehr reviewed Jun 15, 2026

View reviewed changes

Comment thread lib/TaskProcessing/AudioToAudioTranslateProvider.php Outdated

marcelklehr reviewed Jun 15, 2026

View reviewed changes

Comment thread lib/TaskProcessing/AudioToAudioTranslateProvider.php

marcelklehr reviewed Jun 15, 2026

View reviewed changes

Comment thread lib/TaskProcessing/AudioToAudioTranslateProvider.php Outdated

lukasdotcom reviewed Jun 15, 2026

View reviewed changes

julien-nc force-pushed the enh/noid/audio-translation branch from 9181c87 to d173c96 Compare June 15, 2026 14:28

marcelklehr reviewed Jun 15, 2026

View reviewed changes

Comment thread lib/TaskProcessing/AudioToAudioTranslateProvider.php Outdated

julien-nc requested review from lukasdotcom June 15, 2026 14:59

marcelklehr approved these changes Jun 15, 2026

View reviewed changes

feat(audio-translation): add audio translation task type and provider…

4c32176

…, factorize translation logic in the translate service Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

julien-nc added 6 commits June 16, 2026 12:11

adjust tests

18e3819

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

update composer config.platform, update compose dependencies

e5eebc7

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

run cs:fix

1963f7d

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

include audio input transcription as optional output

3f55524

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

include optional inputs: voice, model and speed

b77c2a9

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

watermark the audio transcript so the audio and text outputs are wate…

e785e27

…rmarked in the target language Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

julien-nc force-pushed the enh/noid/audio-translation branch from 3d348ab to 17df89a Compare June 16, 2026 10:47

julien-nc added 2 commits June 16, 2026 13:01

add basic test for AudioToAudioTranslateProvider

d01c43f

Signed-off-by: Julien Veyssier <julien-nc@posteo.net>

julien-nc force-pushed the enh/noid/audio-translation branch from a631631 to d01c43f Compare June 16, 2026 11:01

julien-nc merged commit 28f8da9 into main Jun 16, 2026
26 checks passed

julien-nc deleted the enh/noid/audio-translation branch June 16, 2026 11:38

Uh oh!

Conversation

julien-nc commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

julien-nc commented Jun 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lukasdotcom left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcelklehr commented Jun 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

julien-nc commented Feb 4, 2026 •

edited

Loading