Understanding CockroachDB Replicas

Question

I"m having hard time understanding why the CockroachDB admin console my single node setup has 37 replicas. Based on what I've read

CockroachDB replicates each range (3 times by default) and stores each replica on a different node.

This is directly from the docs https://www.cockroachlabs.com/docs/v20.2/architecture/overview#glossary

Running the command l I see

  database_name
-----------------
  defaultdb
  postgres
  system
  test2
(4 rows)

Running the command SHOW ALL ZONE CONFIGURATIONS; I get

target                      |                               raw_config_sql
---------------------------------------------------+------------------------------------------------------------------------------
  RANGE default                                    | ALTER RANGE default CONFIGURE ZONE USING
                                                   |     range_min_bytes = 134217728,
                                                   |     range_max_bytes = 536870912,
                                                   |     gc.ttlseconds = 90000,
                                                   |     num_replicas = 3,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  DATABASE system                                  | ALTER DATABASE system CONFIGURE ZONE USING
                                                   |     range_min_bytes = 134217728,
                                                   |     range_max_bytes = 536870912,
                                                   |     gc.ttlseconds = 90000,
                                                   |     num_replicas = 5,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  RANGE meta                                       | ALTER RANGE meta CONFIGURE ZONE USING
                                                   |     range_min_bytes = 134217728,
                                                   |     range_max_bytes = 536870912,
                                                   |     gc.ttlseconds = 3600,
                                                   |     num_replicas = 5,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  RANGE system                                     | ALTER RANGE system CONFIGURE ZONE USING
                                                   |     range_min_bytes = 134217728,
                                                   |     range_max_bytes = 536870912,
                                                   |     gc.ttlseconds = 90000,
                                                   |     num_replicas = 5,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  RANGE liveness                                   | ALTER RANGE liveness CONFIGURE ZONE USING
                                                   |     range_min_bytes = 134217728,
                                                   |     range_max_bytes = 536870912,
                                                   |     gc.ttlseconds = 600,
                                                   |     num_replicas = 5,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  TABLE system.public.replication_constraint_stats | ALTER TABLE system.public.replication_constraint_stats CONFIGURE ZONE USING
                                                   |     gc.ttlseconds = 600,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
  TABLE system.public.replication_stats            | ALTER TABLE system.public.replication_stats CONFIGURE ZONE USING
                                                   |     gc.ttlseconds = 600,
                                                   |     constraints = '[]',
                                                   |     lease_preferences = '[]'
(7 rows)

I'm not sure where the 37 comes from, should it not just be 3 since I only created the database test2? Or If it replicates even the default databases it will still only be 3*4 = 12? None of my databases exceed 512M so at most it should only take 1 range for each database. I must be misunderstanding something, can someone give me a hand? Thank you.

question from:https://stackoverflow.com/questions/65943748/understanding-cockroachdb-replicas

Answer 1

Cockroach has a relatively large number of internal system ranges. It maintains internal system tables as well as other bootstrapping metadata. Cockroach splits ranges on table boundaries as well as some other hard-coded split points. You can discover the set of ranges by running a query like:

> SELECT start_pretty, end_pretty, database_name, table_name, replicas FROM crdb_internal.ranges_no_leases;

          start_pretty          |          end_pretty           | database_name |           table_name            | replicas
--------------------------------+-------------------------------+---------------+---------------------------------+-----------
  /Min                          | /System/NodeLiveness          |               |                                 | {1,2,3}
  /System/NodeLiveness          | /System/NodeLivenessMax       |               |                                 | {1,2,3}
  /System/NodeLivenessMax       | /System/tsd                   |               |                                 | {1,2,3}
  /System/tsd                   | /System/"tse"                 |               |                                 | {1,2,3}
  /System/"tse"                 | /Table/SystemConfigSpan/Start |               |                                 | {1,2,3}
  /Table/SystemConfigSpan/Start | /Table/11                     |               |                                 | {1,2,3}
  /Table/11                     | /Table/12                     | system        | lease                           | {1,2,3}
  /Table/12                     | /Table/13                     | system        | eventlog                        | {1,2,3}
  /Table/13                     | /Table/14                     | system        | rangelog                        | {1,2,3}
  /Table/14                     | /Table/15                     | system        | ui                              | {1,2,3}
  /Table/15                     | /Table/16                     | system        | jobs                            | {1,2,3}
  /Table/16                     | /Table/17                     |               |                                 | {1,2,3}
  /Table/17                     | /Table/18                     |               |                                 | {1,2,3}
  /Table/18                     | /Table/19                     |               |                                 | {1}
  /Table/19                     | /Table/20                     | system        | web_sessions                    | {1,2,3}
  /Table/20                     | /Table/21                     | system        | table_statistics                | {1,2,3}
  /Table/21                     | /Table/22                     | system        | locations                       | {1,2,3}
  /Table/22                     | /Table/23                     |               |                                 | {1,2,3}
  /Table/23                     | /Table/24                     | system        | role_members                    | {1,2,3}
  /Table/24                     | /Table/25                     | system        | comments                        | {1,2,3}
  /Table/25                     | /Table/26                     | system        | replication_constraint_stats    | {1,2,3}
  /Table/26                     | /Table/27                     | system        | replication_critical_localities | {1,2,3}
  /Table/27                     | /Table/28                     | system        | replication_stats               | {1,2,3}
  /Table/28                     | /Table/29                     | system        | reports_meta                    | {1}
  /Table/29                     | /NamespaceTable/30            |               |                                 | {1,2,3}
  /NamespaceTable/30            | /NamespaceTable/Max           | system        | namespace2                      | {1,2,3}
  /NamespaceTable/Max           | /Table/32                     | system        | protected_ts_meta               | {1,2,3}
  /Table/32                     | /Table/33                     | system        | protected_ts_records            | {1,2,3}
  /Table/33                     | /Table/34                     | system        | role_options                    | {1,2,3}
  /Table/34                     | /Table/35                     | system        | statement_bundle_chunks         | {1,2,3}
  /Table/35                     | /Table/36                     | system        | statement_diagnostics_requests  | {1,2,3}
  /Table/36                     | /Table/37                     | system        | statement_diagnostics           | {1}
  /Table/37                     | /Table/38                     | system        | scheduled_jobs                  | {1,2,3}
  /Table/38                     | /Table/39                     |               |                                 | {1,2,3}
  /Table/39                     | /Max                          | system        | sqlliveness                     | {1,2,3}
(35 rows)

Each entry in the replicas field represents a replica. Hope this gives you some insights into what ranges exist.