manus-skills/.kortix/opencode/skills/coding/aws-serverless/references/troubleshooting.md

Serverless Troubleshooting Reference

Error → cause → fix, focused on non-obvious gotchas and exact CLI commands. The five most common fixes are first.

Quick fixes

• 502 Bad Gateway (proxy): Lambda response must be { statusCode: int, headers: {}, body: "string" }. body must be a JSON.stringify'd string, not an object. The function ran fine — only the response shape was wrong. (String statusCode is coerced. Under HTTP API payload format 2.0 a missing statusCode defaults to 200, but under REST proxy a missing statusCode is itself a malformed response and triggers the 502.)
• CORS errors: under proxy integration the Lambda returns CORS headers on every response including errors; the console "Enable CORS" button doesn't apply. HTTP API: use CorsConfiguration. CORS is browser-enforced — the API call itself succeeded. Never use AllowOrigin: '*' in production — wildcard CORS is a security risk (CWE-942); specify your exact domain (e.g. https://app.example.com).
• 504 from API Gateway: integration timeout (REST 29s default; HTTP 30s max), independent of Lambda's 15-min limit. For long ops: return 202 immediately, process via SQS/Step Functions, poll or use WebSocket.
• VPC Lambda can't reach internet: needs private subnet + NAT Gateway in a public subnet. A public subnet does NOT give Lambda a public IP. For AWS services use VPC endpoints (free for S3/DynamoDB gateway endpoints).
• ImportModuleError / MODULE_NOT_FOUND: handler path doesn't match file structure, or deps weren't bundled for the right platform. Python: pip install -r requirements.txt -t ./package --platform manylinux2014_x86_64 --only-binary=:all:. Use sam build for cross-platform packaging.

---

Lambda errors (gotchas)

Function URL 403 Forbidden (AuthType=NONE)

A public Function URL returns 403 Forbidden even with AuthType=NONE when the resource-based policy grants only lambda:InvokeFunctionUrl. Invoking a Function URL always requires both lambda:InvokeFunctionUrl and lambda:InvokeFunction — the console and SAM add both statements automatically, but with the CLI/API you must add each yourself (two separate add-permission calls):

# Statement 1: allow invoking via the Function URL (public)
aws lambda add-permission --function-name my-function \
  --statement-id FunctionURLAllowPublicAccess \
  --action lambda:InvokeFunctionUrl --principal '*' \
  --function-url-auth-type NONE

# Statement 2: REQUIRED — allow the underlying invoke, scoped to URL calls
aws lambda add-permission --function-name my-function \
  --statement-id FunctionURLInvokeAllowPublicAccess \
  --action lambda:InvokeFunction --principal '*' \
  --invoked-via-function-url

The --invoked-via-function-url flag sets the lambda:InvokedViaFunctionUrl condition on statement 2, scoping it to Function URL calls so it does not open direct Invoke access. This 403 is not an org SCP/RCP block — Lambda does not support RCPs, so don't chase that path. See Function URLs and Lambda function URL auth.

RecursiveInvocationException

A function writes to a resource that re-triggers it (Lambda halts after ~16 loops).

# Emergency stop:
aws lambda put-function-concurrency --function-name my-func --reserved-concurrent-executions 0
# Fix: separate input/output buckets, or prefix filters in the trigger config

SnapStart errors

SnapStartException / SnapStartNotReadyException / SnapStartTimeoutException — init threw, or it uses non-snapshottable resources (open network connections).

aws lambda get-function --function-name my-func --query 'Configuration.SnapStart'
# Java:   CRaC hooks — beforeCheckpoint() closes connections, afterRestore() reopens
# Python: snapshot_restore runtime hooks to re-establish connections after restore
# .NET:   SnapshotRestore.Register hooks for before-snapshot / after-restore

Sandbox.Timedout

Exceeded the timeout. In newer runtimes this covers both init-phase and invoke-phase timeouts — a suppressed init failure consumes the invoke timeout. Move heavy init to lazy loading inside the handler; raise --timeout/--memory-size.

ENILimitReachedException

VPC hit its network-interface quota. Two limits can apply — the Lambda-specific Hyperplane ENI-per-VPC quota and the broader VPC ENI-per-Region quota — so check which one was reached (Service Quotas / describe-account-attributes).

aws service-quotas request-service-quota-increase --service-code vpc --quota-code L-DF5E4CA3 --desired-value 10000
# Consolidate functions onto the same subnet + security group combinations

TooManyRequestsException / throttling

aws lambda get-account-settings
aws service-quotas request-service-quota-increase --service-code lambda --quota-code L-B99A9384 --desired-value 3000
aws lambda put-function-concurrency --function-name my-func --reserved-concurrent-executions 100

Throttle behavior by invocation type: Sync (API GW) → 429 (may surface as 500); Async (S3, SNS) → auto-retries up to 6h; SQS → returns to queue, backs off; Kinesis/DynamoDB Streams → retries batch, blocks shard.

CodeStorageExceededException

Account exceeded its per-Region code storage quota (unzipped; all versions + layers). Check the current limit with aws service-quotas get-service-quota --service-code lambda --quota-code L-2ACBD22F (it is adjustable and has changed over time). Delete old versions and unused layers (aws lambda list-versions-by-function, delete-function --qualifier).

---

API Gateway errors (gotchas)

Error	Cause	Fix
403 Missing Authentication Token	URL doesn't match a resource/method, or API not deployed	Usually routing, not auth — verify path and create-deployment to the stage
Invalid permissions on Lambda function (500)	API GW lacks lambda:InvokeFunction	aws lambda add-permission --principal apigateway.amazonaws.com --source-arn "arn:aws:execute-api:...:api-id/*/GET/resource" (add --source-account <account-id> to scope cross-service triggers like S3 to your account)
Unauthorized (401)	Lambda authorizer denied/errored/timed out	Check authorizer logs; verify it returns a valid Allow policy
403 + x-amzn-errortype: ForbiddenException	WAF rule matched	Check WAF sampled requests; test rules in Count mode first
500 with log status 429	Lambda throttled, surfaced as 500	Increase Lambda concurrency

---

Step Functions errors

Error	Cause / Fix
States.TaskFailed	Add Retry for States.TaskFailed/Lambda.ServiceException/Lambda.SdkClientException; Catch to a handler
States.Timeout	Set TimeoutSeconds (and HeartbeatSeconds for long tasks)
States.DataLimitExceeded	Input/output > 256 KiB — cannot be caught by States.ALL. Store in S3, pass keys
ExecutionAlreadyExists	Names unique per state machine for 90 days — use --name "exec-$(date +%s)" or omit --name
States.Permissions	Execution role lacks target-service permission

---

SAM/CDK errors

Stale build cache

sam build --no-cached        # correct flag — there is NO --clear-cache
rm -rf .aws-sam/cache        # or delete the cache directory

PythonPipBuilder:ResolveDependencies

Version conflicts or missing native libs: sam build --use-container --no-cached; use binary wheels (psycopg2-binary).

CDK NodejsFunction: Cannot find module 'esbuild'

npm install --save-dev esbuild.

CREATE_FAILED / UPDATE_ROLLBACK_FAILED

aws cloudformation describe-stack-events --stack-name my-stack \
  --query "StackEvents[?ResourceStatus=='CREATE_FAILED'].[LogicalResourceId,ResourceStatusReason]" --output table
# Rollback also failed (resource manually deleted / perms changed):
aws cloudformation continue-update-rollback --stack-name my-stack --resources-to-skip MyFunction

Circular dependency

Circular dependency between resources: [MyFunction, MyRole, ...] — break the cycle by giving the function an explicit FunctionName and referencing the hardcoded ARN (!Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:my-function-name") instead of !Ref/!GetAtt, or extract the IAM role/policy into a separate resource.

Other quick ones

• DockerBuildFailed → docker info / start Docker.
• Security Constraints Not Satisfied (missing Handler/Runtime/CodeUri) → sam validate --lint.
• This stack uses assets, so the toolkit stack must be deployed → cdk bootstrap aws://ACCOUNT/REGION.

---

Diagnostics

For systematic timeout investigation (config → logs → metrics → VPC → cold start → memory → downstream), use the debugging-lambda-timeouts skill (see SKILL.md routing).

CloudWatch Logs Insights run against /aws/lambda/FUNCTION_NAME (API Gateway uses the access log group). Useful fields on @type = "REPORT" lines: @initDuration (cold start), @duration, @billedDuration, @maxMemoryUsed, @memorySize. Examples:

# Cold start rate + init latency
filter @type="REPORT" | stats count() as total, sum(ispresent(@initDuration)) as coldStarts,
  avg(@initDuration) as avgInitMs, pct(@initDuration,99) as p99InitMs by bin(1h)

# Out-of-memory (>90% used)
filter @type="REPORT" | filter @maxMemoryUsed/@memorySize > 0.9
  | fields @timestamp, @requestId, @maxMemoryUsed/1e6 as usedMB | sort @timestamp desc

# Latency percentiles
filter @type="REPORT" | stats pct(@duration,50) as p50, pct(@duration,90) as p90,
  pct(@duration,99) as p99, max(@duration) as max by bin(1h)

# API Gateway 5xx (access log group)
filter status >= 500 | stats count() as errors by status, path, httpMethod | sort errors desc

Memory ↔ vCPU (for OOM / CPU tuning)

Memory	vCPUs	Use case
128 MB	~0.08	Simple transforms
512 MB	~0.3	Moderate processing
1,769 MB	1.0	CPU-intensive single-threaded
3,538 MB	2.0	Multi-threaded
10,240 MB	~5.8 (up to 6 vCPU)	Heavy compute / ML inference

X-Ray

Enable via SAM Tracing: Active (Function) / TracingEnabled: true (Api) or CDK tracing: lambda.Tracing.ACTIVE (adds AWSXRayDaemonWriteAccess automatically). Default sampling: 1 req/s reservoir + 5%. Instrument SDK calls with patch_all() (Python) / captureAWSv3Client (Node).