c# - How to handle changes from different containers in Cosmos DB through Change Feed

Question

I have some amount of containers in Cosmos DB that changes all the time. I need to provide some mechanism for reading all the changes from those containers.

I'm trying to implement builder/factory for Change Feed Processor (CFP). In my case, I have to create CFP instances dynamically for the different container. How I see the solution right now - I need a WebJob/Console Application that listens to the queue. When another application creates a new container in Cosmos DB it also sends a new message to the queue. Message in the queue contains all information for creating new CFP (connection string, collection name, lease container name, etc). The application creates new CFP and runs it in a new thread in the background forever.

Here is the code how I'm creating a new CFP

private void StartNewProcessor()
{
    new List<Task>().Add(Task.Run(async () =>
    {
        var container = Database.GetContainer(ContainerName);
        var lease = Database.GetContainer(LeaseName);

        var changeFeedProcessor = container.GetChangeFeedProcessorBuilder<Item>(ProcessorName, ProcessData)
            .WithLeaseContainer(lease)
            .WithInstanceName(InstanceName)
            .Build();

        await changeFeedProcessor.StartAsync();

        Console.WriteLine($"Change Feed Processor: {ProcessorName} have been started");

        Console.ReadKey(true);

        await changeFeedProcessor.StopAsync();
    }));
}

The problem is that it's a bad approach since there can be 100 and more collections in the future, so I'll need to create 100 extra threads in background. I'm looking for some ideas regarding the architecture application and how to do all that in the right way. It will be great if it is possible to handle changes for all containers in one application.

question from:https://stackoverflow.com/questions/65922236/how-to-handle-changes-from-different-containers-in-cosmos-db-through-change-feed

Answer 1

Waitting for answers