While developing with Google Vertex AI’s Generative AI models, API requests suddenly started failing. In this post, I’ll share the error code 429 I encountered and the solution that resolved it.

The Problem

When sending requests from our application to the Vertex AI endpoint, we received the following error message:

Failed after 3 attempts. Last error: Resource exhausted. Please try again later. Please refer to https://cloud.google.com/vertex-ai/generative-ai/docs/error-code-429 for more details.

The error code was 429 with a “Resource exhausted” message. Initially, I thought it was simply a request rate limit issue and tried retrying after waiting, but the situation didn’t improve.

Investigation and Root Cause Analysis

Referencing the URL included in the error message, I checked the official documentation and found that error code 429 occurs due to one of the following reasons:

1. Project request rate exceeds quota limits.
2. Temporary resource shortage in a specific region.

In our case, the request volume wasn’t particularly high, so I determined that the latter “resource shortage in a specific region” was more likely. Our environment was configured to send API requests to the us-central1 region.

Solution: Changing Location to global

I decided to change the API request endpoint location. Specifically, I modified the environment variable GOOGLE_VERTEX_LOCATION configured in our application as follows:

Before:

GOOGLE_VERTEX_LOCATION="us-central1"

After:

GOOGLE_VERTEX_LOCATION="global"

After deploying this change, the frequently occurring error code 429 stopped happening completely, and API requests began succeeding consistently.

Why Did global Resolve the Issue?

According to the official documentation, Vertex AI generative AI models have both regional endpoints like us-central1 and a global endpoint.

• Regional endpoints (us-central1, etc.): Requests are processed by physical computing resources within that region.
• Global endpoint (global): Requests are routed to the nearest available region based on the sender’s location.

In our case, it’s likely that the us-central1 region was experiencing temporary high load, causing resource exhaustion. By switching to the global endpoint, Google can dynamically distribute traffic to other available regions, thereby avoiding the resource exhaustion problem.

Summary

When encountering error code 429 (Resource exhausted) with Google Vertex AI, implementing retry logic is certainly important, but reviewing the API request location settings is also a very effective approach.

If you’re specifying a particular region and facing similar errors, I recommend trying the global location.

That’s all from the Gemba.

Reference Information

• Error code 429  |  Generative AI on Vertex AI  |  Google Cloud
• Google model endpoint locations  |  Generative AI on Vertex AI  |  Google Cloud