optimization - SOLR index size reduction

Question

We have a some massive SOLR indices for a large project, and its consuming above 50 GB of space .

We have considered several ways to reduce the size that are related to changing the content in the indices, but I am curious of wether or not there might be any changes we can make to a SOLR index which will reduce its size by 2 orders of magnitude or more, which are directly related to either (1) maintainance commands we can run or (2) simple configuration parameters which may not be set right.

Another relevant question is (3) Is there a way to trade index size for performance inside of SOLR, and if so , how would it work ?

Any thoughts on this would be appreciated... Thanks!

Answer 1

There are a couple things you might be able to do to trade performance for index size. For example, an integer (int) field uses less space than a trie integer (tint), but range queries will be slower when using an int.

To make major reductions in your index, you will almost certainly need to look more closely at the fields you are using.

• Are you using a lot of stored fields? If so, try removing the stored fields from the index and query your database for the necessary data once you've got the results back from Solr.
• Add omitNorms="true" to text fields that don't need length normalization
• Add omitPositions="true" to text fields that don't require phrase matching
• Special fields, like NGrams, can take up a lot of space
• Are you removing stop words from text fields?