Debugging 403 Permission Failures in Event-Driven GCP Systems
A 403 is an authorization decision with evidence behind it. The fastest safe fix is to identify the exact principal, resource and permission at the denied hop, then test the narrowest binding that satisfies the intended flow.
Draw the call graph
List each hop: scheduler to target, publisher to topic, push subscription to service, service to task queue, and worker to downstream API. Attach a concrete identity to every arrow. The identity configured on the resource may differ from the one shown in the denial.
Read the denial as data
Use Cloud Audit Logs and the service's request logs to capture principalEmail, permission, resource name, service and status detail. Confirm project and region. A similarly named service account in another project is a common source of confusion.
Check both caller and service agents
Some managed services use Google-managed service agents in addition to the workload service account. Verify that the relevant service API is enabled and that its service agent has the documented role. Do not replace missing service-agent access with broad user privileges.
Grant at the narrowest useful scope
Prefer a predefined role on the specific topic, queue, service or secret. Avoid basic Editor and Owner roles. Where a role contains more permissions than needed, document the gap and consider a custom role only when its maintenance cost is justified.
Reproduce with explicit credentials
Impersonate the suspected caller where authorised and invoke the same operation against a non-production target. This distinguishes token audience, IAM and application authorization problems. Account for IAM propagation before changing the policy again.
Prevent recurrence
Codify bindings in Terraform, test critical permissions in CI and alert on IAM drift. Record the causal permission and principal in the incident note so future responders do not repeat a trial-and-error privilege escalation.