c# - How to avoid memory exception when reading, creating and sending a very large xml file?

Question

I am working on to optimize our code so that we can read, create and send xml file which can be very large in size (2GB).

For read and create we are using XmlReader class.

We actually get an XML string from some other service. If we store the XML string in a string variable it takes the same amount of memory. That point aside, please suggest the best way to deal with the XML string so the memory out of bound exception doesn't occur.

I can not show code over here due to company policies but that should not matter because code is already working but in case of large xml string its giving:

memory exception

...as mentioned.

EXPLANATION :

We get 2GB Xml from a service.

We process it using streaming.

Since we need to read that xml using xmlreader, we pass the xml in the form of string to create a new xml with almost same size (2GB)

byte[] msg = Buffer.ExtractMessage(messageStart, messageEnd);  
string msg1 = Encoding.UTF8.GetString(msg);

CreateNewXMLFileFromTheCurrentXmlString(msg1);

We then send that new xml to some other service.

Answer 1

The best way would be to use a well normalized and indexed database if that's possible for you. Then getting the data by using LINQ should solve your problems. And the problem is the source and not your logic as XML files shouldn't be as big as yours.

Take a look here: LINQ TO XML