ECS Deployment Fixes

Calliope Integration: This component is integrated into the Calliope AI platform. Some features and configurations may differ from the upstream project.

This document describes fixes for issues identified in the ECS deployment of WAIIDE/Calliope WAIIDE.

Issues Identified

1. Missing WAIIDE Extensions

Multiple extensions were failing to load due to missing package.json files:

calliope-WAIIDE-agent
git, github, github-authentication
markdown-language-features
merge-conflict
microsoft-authentication
typescript-language-features

2. 404 Errors on /api Endpoint

Continuous 404 errors for GET requests to /api from health checks.

3. JupyterHub OAuth Issues

HTTP 403 Forbidden errors when notifying Hub of activity
OAuth redirect URLs incorrectly constructed
Missing OAuth scopes for named servers

4. Missing Marketplace Configuration

The file /opt/calliope/assets/product-patch.json was missing, preventing marketplace access.

5. User Experience Issues

WAIIDE and terminals starting in /opt/calliope instead of user’s workspace
File browser not rooted in user’s directory

Fixes Applied

1. Enhanced Extension Copying (`scripts/fix-extension-copy.sh`)

Created a dedicated script that:

Safely copies complete extension directories with verification
Handles missing package.json files by re-copying from source
Supports multiple source locations (template, WAIIDE-server, with prefixes)
Provides detailed logging and verification

2. Product Patch for Marketplace (`assets/product-patch.json`)

Created comprehensive marketplace configuration to enable WAIIDE extension gallery:

Full marketplace URLs and endpoints
Authentication configuration for Microsoft services
Trusted domains for extension downloads
Built-in extension declarations

3. Runtime Marketplace Configuration

Keep jq installed: Modified Dockerfile to keep jq in runtime image
Runtime patching: Entrypoint applies marketplace config at startup
Verification: Checks for marketplace URLs after patching
Fallback: Python-based patching if jq unavailable

4. User Workspace Configuration

Workspace directory: Set to /home/$JUPYTERHUB_USER/workspace
WAIIDE default folder: Starts in user’s workspace
Terminal working directory: Configured to start in workspace
Jupyter root directory: Set to user’s workspace
Default settings: Applied to ensure consistent user experience

5. Default WAIIDE Settings (`assets/default-settings.json`)

Created default settings for better user experience:

Terminal starts in user workspace
File explorer rooted in workspace
Auto-save enabled
Pergamon theme as default
Telemetry disabled
Marketplace extensions enabled

3. Jupyter Server API Handlers (`scripts/jupyter_server_config.py`)

Added custom handlers for API endpoints:

/api - Returns API information
/api/status - Health check endpoint
Prevents 404 errors during health checks

4. Improved Entrypoint Script (`scripts/entrypoint-jupyterhub.sh`)

Updated to:

Call the extension fix script during setup
Better handling of extension copying with verification
Improved error recovery for missing extensions

5. OAuth Authentication Fixes

The existing OAuth fix modules (oauth_named_server_fix.py, jupyter_scope_fix.py) handle:

Correct OAuth redirect URLs for named servers
Proper scope validation for named server access
Permissive authentication for JupyterHub users

Deployment Steps

Build New Docker Image
```
docker build -t calliope-waiide:fixed .
```
Update ECS Task Definition Update the task definition to use the new image with fixes.
Verify Fixes After Deployment
- Check that all extensions load without errors
- Verify /api/status returns 200 OK
- Confirm OAuth authentication works without 403 errors
- Test WAIIDE marketplace access

Testing Checklist

WAIIDE extensions load without package.json errors
Health checks on /api/status return 200 OK
No 404 errors for /api endpoint
JupyterHub activity notifications succeed
OAuth authentication works for named servers
WAIIDE marketplace is accessible
Users can install extensions from marketplace

Monitoring

After deployment, monitor logs for:

Extension loading errors
API endpoint 404s
OAuth 403 errors
JupyterHub activity notification failures

Rollback Plan

If issues persist:

Revert to previous ECS task definition
Review CloudWatch logs for new errors
Test fixes locally with Docker Compose
Apply additional fixes and redeploy

Future Improvements

Pre-build Validation: Add tests to verify all extensions have valid package.json files before building image
Health Check Enhancement: Add more comprehensive health checks for all components
Extension Caching: Implement extension caching to speed up container startup
OAuth Token Refresh: Implement automatic OAuth token refresh for long-running sessions

Documentation Cleanup Summary - July 2025 ECS Instance Names for WAIIDE