Command aggregate failed: Exceeded memory limit for $group, but didn't allow external sort

mongo
mongo
170 Points
8 Posts

I have written group aggregate pile line. It was working fine. But now getting error as:

Command aggregate failed: Exceeded memory limit for $group, but didn't allow external sort. Pass allowDiskUse:true to opt in..MongoDB.Driver.Core!

I'm trying something:

var group = new BsonDocument
        {
                { "_id", "$_id" },
                { "root", new BsonDocument{ { "$mergeObjects", "$$ROOT" } } },
                {  "items", new BsonDocument{ { "$push", "$items" } } }
        };

var orders = mongoDBService.GetCollection("orders").Aggregate()
                            .Lookup("items", "items.itemId", "_id", @as: "items")
                            .Unwind("items", new AggregateUnwindOptions<ItemDetail>() { PreserveNullAndEmptyArrays = true })
                            .Lookup("vendors", "items.vendorId", "_id", @as: "items.vendor")
                            .Unwind("items.vendor", new AggregateUnwindOptions<VendorDetail>() { PreserveNullAndEmptyArrays = true })
                            .Group(group)
                            .ReplaceRoot<object>("{$mergeObjects:['$root', '$$ROOT']}")
                            .Project("{root:0}")
                            .As<OrderDetail>().ToEnumerable();
Views: 8615
Total Answered: 2
Total Marked As Answer: 1
Posted On: 12-Oct-2021 02:43

Share:   fb twitter linkedin
Answers
Smith
Smith
2830 Points
78 Posts
         

Set options in aggregation as mentioned in the error message:

new AggregateOptions() { AllowDiskUse = true }

as

var group = new BsonDocument
        {
                { "_id", "$_id" },
                { "root", new BsonDocument{ { "$mergeObjects", "$$ROOT" } } },
                {  "items", new BsonDocument{ { "$push", "$items" } } }
        };

var orders = mongoDBService.GetCollection("orders").Aggregate(new AggregateOptions() { AllowDiskUse = true })
                            .Lookup("items", "items.itemId", "_id", @as: "items")
                            .Unwind("items", new AggregateUnwindOptions<ItemDetail>() { PreserveNullAndEmptyArrays = true })
                            .Lookup("vendors", "items.vendorId", "_id", @as: "items.vendor")
                            .Unwind("items.vendor", new AggregateUnwindOptions<VendorDetail>() { PreserveNullAndEmptyArrays = true })
                            .Group(group)
                            .ReplaceRoot<object>("{$mergeObjects:['$root', '$$ROOT']}")
                            .Project("{root:0}")
                            .As<OrderDetail>().ToEnumerable();

References:

Posted On: 12-Oct-2021 22:20
Thanks. Works for me.
 - mongo  17-Oct-2021 22:57
Rahul Maurya
Rahul M...
4896 Points
25 Posts
         

Aggregation pipeline stages have maximum memory uses limit in mongodb. To handle large datasets, you are required to set allowDiskUse option to true to enable writing data to temporary files in disk storage. Aggregate like sort, group have a maximum of RAM configured, but when exceeded instead of abort the operation it will continue using disk file storage instead of RAM, if the allowDiskUse has been set to true.

db.getCollection('orders').aggregate( [ { $sort : { created : 1} } ], { allowDiskUse: true } )

 

Note: Data in memory is faster than working with data on hard drive. With hard drive data transfer the aggregation operation will need to keep transferring data between the operation, memory and the hard disk.

 

Posted On: 17-Oct-2021 22:43
 Log In to Chat