Cassandra read latency increases while writing

Question

I have a cassandra cluster, its read latency increases during writes. The writes mostly happen via spark jobs during the night time. The writes happen in huge bursts, is there a way to reduce read latency during the writes. The writes happen using LOCAL_QUORUM and reads happen using LOCAL_ONE. Is there a way to reduce read latency when writes are happening?

Cassandra Cluster Configs

10 Node cassandra cluster (5 in DC1, 5 in DC2)

CPU: 8 Core

Memory: 32GB

Grafana Metrics

Answer 1

I can give some advice:

1. Use LCS compaction strategy.

2. Prefer round-robin load balancing policy for reads.

3. Choose partition_key wisely so that requests are not bombarded on a single partition.

Partition size also play a good role. Cassandra recommends to have smaller partition size. However, I have tested with Partitions of 10000 rows each with each row having size of 800 bytes. It worked better than with 3000 rows(or even 1 row). Very tiny partitions tend to increase CPU usage when data stored is large in terms of row count. However, very large partitions should be avoided even.

4. Replication Factor should be chosen strategically . Write consistency level should be decided considering the replication of all keyspaces.