cosmosdb-error-429 – Large request rate error Cosmos Mongo DB solved

I developed a schedule job to upload health care data in Mongo DB. Our team is using Microsoft Azure Cosmos DB for UAT envronment. I got cosmosdb-error-429 error. Here is the complete error log –

Errors : [ "Request rate is large. More Request Units may be needed, so no changes were made. Please retry this request later. Learn more: http://aka.ms/cosmosdb-error-429"]

Due to this error, I record deletion and insertion also got failed. The schedule job was about to delete all the previous records and insert new records completely. But all in vain due to cosmosdb-error-429 error.

Here is my full console logs :

ERROR [env=t02 appname=poc-mail-api-service vcaprequestid= traceid= spanid= parentspanid= session= authlevel= authusertype= authuser= user= customer= klid=] 15 --- [scheduling-1 ] c.r.w.c.a.s.HealthCodeScheduledTasks : Error while inserting health codes :org.springframework.dao.DataIntegrityViolationException: Write operation error on server poc-cosmos-dev-westeurope.mongo.cosmos.azure.com:10255. Write error: WriteError{code=16500, message='Error=16500, RetryAfterMs=1225, Details='Response status code does not indicate success: TooManyRequests (429); Substatus: 3200; ActivityId: f4a63494-6fe0-4fca-b33b-2711af96f208; Reason: (Errors : [ "Request rate is large. More Request Units may be needed, so no changes were made. Please retry this request later. Learn more: http://aka.ms/cosmosdb-error-429"]);', details={}}.; nested exception is com.mongodb.MongoWriteException: Write operation error on server poc-cosmos-dev-westeurope.mongo.cosmos.azure.com:10255. Write error: WriteError{code=16500, message='Error=16500, RetryAfterMs=1225, Details='Response status code does not indicate success: TooManyRequests (429); Substatus: 3200; ActivityId: f4a63494-6fe0-4fca-b33b-2711af96f208; Reason: (Errors : [ "Request rate is large. More Request Units may be needed, so no changes were made. Please retry this request later. Learn more: http://aka.ms/cosmosdb-error-429"]);', details={}}.: ex:{}

Due to this error sometimes I also observed duplicate records inserted. Sometimes records did not delete. I mean that inconsistency in delete and insertion DB operations.

Main Reason for cosmosdb-error-429

I did a bit analysis to find root cause for this issue. Because root cause always helps to give permanent solution. Basic problem for this issue is Cosmos DB request units allowed per second.

So the collection object in request exceeded the throughput limit. This is the root casue. Generally it happens when more than one large request made on DB.

For our Cosmos DB 400 RU/s was allowed. We need more request units to overcome this problem. But budget is also one constraint for us.

Solution for Large request rate error

So to overcome cosmosdb-error-429 related to large request rate error there is 2 solutions.

Solution by increasing request units in Cosmos DB

First solution is to increase the RU/s. For that you need to go to “scale & settings” section. You can select manual option and increase RU/s from 400 RU/s to 600 RU/s.

I recommend you to analyse your request size and then decide appropriate request units. Because it is directly related to cost.

Please refer to this Cosmos DB troubleshoot guide for 429 error. This guide will help you to analyse – Request rate is large. More Request Units may be needed error.

Retry logic can solve cosmosdb-error-429 – Request rate is large. More Request Units may be needed

 @Scheduled(fixedRate = 1000 * 60 * 60)
    public void saveHealthCodesUsingJsonFile() {
        try {

            /**
             * Your data insertion and deletion code
             */   
        
        } catch (InterruptedException exception) {
            log.error("Thread is interrupted!", exception);
            Thread.currentThread().interrupt();
            delayRequestForMongoDBReadiness();
            saveHealthCodesUsingJsonFile();
        } catch (IOException | DataIntegrityViolationException | MongoWriteException | UncategorizedMongoDbException |
                MongoClientException | ExecutionException exception) {
            log.error("Execution error", exception.getMessage());
            delayRequestForMongoDBReadiness();
            saveHealthCodesUsingJsonFile();
        }
    }

    private void delayRequestForMongoDBReadiness() {
        try {
            TimeUnit.SECONDS.sleep(30);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }

In the above code we handle the error by using delayRequestForMongoDBReadiness(); method. You can also delay your request to overcome cosmosdb-error-429 error.

Conclusion

So now you should be able to fix cosmosdb-error-429 error. Request rate is large. More Request Units may be needed error.

In case you need any assistance please drop a message to our team.