Orchestration

Quick links: Public API • Functions API

Overview

Orchestration fans out a post submission to provider‑specific publish functions using Cloud Tasks. Two orchestrators exist: publish (multi‑target) and repost (opt‑in destinations). Media may be normalized once (e.g., video rehost) before fan‑out.

Publish Orchestrator

Entry: apps/functions/orchestrators/publish-orchestrator.js Core logic: apps/functions/orchestrators/publishing-service.js

• Input: { userId, postSubmissionId, request } (from Public API /v1/posts)
• Target resolution: from request.post.content.platform normalized to lowercase; twitter → x.

Account Resolution

Multi-account support is built into the orchestration layer:

• If user has multiple accounts for a platform, accountId is required in the request
• If user has exactly one account, it's automatically selected
• If no connected accounts exist, orchestration fails immediately with missing_integrations error
• Account metadata is validated before task dispatch:
  • integrations/{platform}/{accountId} must exist
  • accessToken and required credentials must be present
  • Account must not be in error state

Media Normalization

Before platform fan-out, media may be normalized to meet platform requirements:

• Video rehosting: Videos are rehosted to Cloud Storage to ensure all platforms can access the media URL
  • Original video is fetched and streamed to a new Cloud Storage object
  • New public URL is generated and used across all platforms
  • Prevents issues with platform-specific download restrictions
• Media validation: File types and sizes are checked against platform limits
• URL accessibility: All media URLs are validated for public accessibility before dispatch

Task Dispatch

For each platform, enqueues a Cloud Task to the provider function:

• facebook → publishFacebook
• instagram → publishInstagram
• threads → publishThreads
• x → publishX
• tiktok → publishTiktok
• youtube → publishYoutube
• linkedin → publishLinkedin

Each task includes:

• userId, postSubmissionId
• Resolved accountId for the platform
• Normalized media URLs
• Platform-specific content (text, hashtags, etc.)
• Distributed trace context (traceId, spanId)

State Management

Writes lifecycle to Firestore:

• postSubmissions/{id}:
  • status: 'orchestrating' → 'dispatched'
  • targets array with resolved platforms and accountIds
  • traceId for distributed tracing
• postSubmissions/{id}/runs/{platform}_{accountId}:
  • status: 'queued'
  • taskName (Cloud Tasks task identifier)
  • accountId (resolved account)
  • spanId (for distributed tracing)
  • Timestamps: createdAt, queuedAt

Distributed Tracing

Every post submission generates a unique traceId that flows through all operations:

• Created at API entry point
• Passed to orchestrator
• Propagated to all platform publish tasks
• Logged in all structured log entries
• Enables end-to-end request tracing across async boundaries

Provider Publish Functions

Each provider validates connectivity under users/{uid}/integrations/{provider}, transitions run state to processing, performs the API call, then writes succeeded or failed with results/errors. On success, a user-visible post record is created under users/{uid}/posts.

References:

• Facebook: integrations/facebook/{auth.js,post.js} — Pages feed posting.
• Instagram: integrations/instagram/{auth.js,post.js} — container + publish for images.
• Threads: integrations/threads/{auth.js,post.js} — text posts.
• X: integrations/x/{auth.js,post.js} — text tweets with PKCE + refresh.
• TikTok: integrations/tiktok/{auth.js,post.js} — OAuth, direct publish for video/photo with status polling.
• YouTube: integrations/youtube/{auth.js,post.js} — OAuth + refresh sweep; publish implemented.
• LinkedIn: integrations/linkedin/{auth.js,post.js} — UGC Posts API for text posting.

Repost Orchestrator

Entry: apps/functions/repost-orchestrator.js

• Triggered by two paths:
  • Firestore: onOrganicPostCreated watches users/{uid}/organicPosts/* for TikTok sources and enqueues repost orchestrator
  • HTTP: pollerTiktok detects newest TikTok video and creates organicPosts docs
• Reads users/{uid}/integrations/{platform}.enabled_for_repost and builds target list
• Enqueues per‑platform repost jobs (via publish mapping) through the same publish fan‑out

Cloud Tasks

Helpers:

• Helper: apps/functions/tasks.js — builds function URLs and enqueues tasks.

Queues:

• orchestrate-posts, publish-*, repost-*. Names are configured via env in shared config.

Queue setup

Cloud Tasks queues are regional and must exist before tasks can be enqueued. Create them in the same region as your Functions (default: us-central1). The names are platform‑agnostic — create only the ones you use today, and add new queues as you onboard more providers.

Basic creation:

REGION=us-central1

# Orchestrator (dispatches per-target publish tasks)
gcloud tasks queues create orchestrate-posts --location="$REGION"

# Provider publish queues (create the ones you need)
gcloud tasks queues create publish-facebook   --location="$REGION"
gcloud tasks queues create publish-instagram  --location="$REGION"
gcloud tasks queues create publish-threads    --location="$REGION"
gcloud tasks queues create publish-x          --location="$REGION"
gcloud tasks queues create publish-tiktok     --location="$REGION"
gcloud tasks queues create publish-youtube    --location="$REGION"

# Example for a new platform (e.g., LinkedIn)
gcloud tasks queues create publish-linkedin   --location="$REGION"

# Optional repost queues (only if you enable repost flows)
gcloud tasks queues create repost-instagram   --location="$REGION"
gcloud tasks queues create repost-youtube     --location="$REGION"

Recommended tuning (optional):

# Example conservative defaults; adjust to your provider rate limits
gcloud tasks queues update publish-x \
  --location="$REGION" \
  --max-dispatches-per-second=10 \
  --max-concurrent-dispatches=5 \
  --max-attempts=5 \
  --min-backoff=10s \
  --max-backoff=600s

Verification:

gcloud tasks queues describe orchestrate-posts --location="$REGION"
gcloud tasks queues describe publish-x        --location="$REGION"

Notes:

• Keep --location aligned with FUNCTION_REGION/deployment region. Queues are regional.
• Our enqueue helper posts unauthenticated HTTP requests to Cloud Functions URLs. If you restrict invocation, extend the task to include an OIDC token bound to a service account with Cloud Functions Invoker and update the function to require auth.
• Queue names are platform‑agnostic. For any new provider, create a publish-<provider> queue and wire it in the orchestrator.

Observability

State Tracking

• Firestore serves as the source of truth for submission and per-platform run states
• users/{uid}/apiRequests logs Public API v1 calls with durationMs and inferred targets
• postSubmissions/{id} tracks orchestration status and target platforms
• postSubmissions/{id}/runs/{platform}_{accountId} tracks individual platform execution

Distributed Tracing

The system implements distributed tracing across all async operations:

• Trace context: Every request generates a traceId (UUID) at the API layer
• Span hierarchy: Each operation (orchestration, platform publish) creates child spans with spanId
• Depth tracking: depth field tracks nesting level (0 = root, 1 = orchestrator, 2 = platform)
• Flow correlation: flowId (typically postSubmissionId) groups related operations
• Structured logs: All log entries include trace context for correlation

Query patterns for debugging:

// Find all logs for a specific post submission
db.collection('logs')
  .where('flowId', '==', postSubmissionId)
  .orderBy('timestamp')

// Find failed operations in a trace
db.collection('logs')
  .where('traceId', '==', traceId)
  .where('spanStatus', '==', 'error')

// Time-series queries with bucketHour
db.collection('logs')
  .where('bucketHour', '>=', '2025-01-15-10')
  .where('bucketHour', '<=', '2025-01-15-12')
  .orderBy('bucketHour')
  .orderBy('timestamp')

See also: Tracing & Logging