linux - Database Killed error while try to start new graph instance on neo4j Desktop

Question

Issue:

Database failed to start, exited with status 'KILLED", Logs end with: Caused by: ... bind(..) failed: Address already in use

Very similar to these:

• Neo4j Desktop DB failed to start with with status 'KILLED'
• https://community.neo4j.com/t/database-failed-to-start/5612
• https://community.neo4j.com/t/database-failed-to-start-db-database-ee8e1c21-23d9-40fd-8a4e-02fd55a0d324-v4-1-0-exited-with-status-killed-check-the-logs/21045

Logs end at:

Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use

Things I've tried:

• Killing all processes associated with neo4j
• searching for default port lsof :11002/3
• reinstalling
  • some config files you don't seem to be able to delete in .config

Os and Version Data

• OS: Linux pop-os 5.4.0-7634-generic #38~1596560323~20.04~7719dbd-Ubuntu SMP Tue Aug 4 19:12:34 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
• neo4j version: 4.1.0
• desktop version: 1.3.4

neo4j.log

Directories in use:
  home:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0
  config:       /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/conf
  logs:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/logs
  plugins:      /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/plugins
  import:       /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/import
  data:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data
  certificates: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/certificates
  run:          /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/run
Starting Neo4j.
WARNING: Max 8192 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
Directories in use:
  home:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0
  config:       /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/conf
  logs:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/logs
  plugins:      /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/plugins
  import:       /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/import
  data:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data
  certificates: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/certificates
  run:          /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/run
Starting Neo4j.
WARNING: Max 8192 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
2020-08-21 19:04:24.399+0000 INFO  Note that since you did not explicitly set the port in dbms.connector.bolt.advertised_address Neo4j automatically set it to 11002 to match dbms.connector.bolt.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO  Note that since you did not explicitly set the port in dbms.connector.http.advertised_address Neo4j automatically set it to 11003 to match dbms.connector.http.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO  Starting...
Directories in use:
  home:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0
  config:       /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/conf
  logs:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/logs
  plugins:      /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/plugins
  import:       /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/import
  data:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data
  certificates: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/certificates
  run:          /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/run
Starting Neo4j.
WARNING: Max 8192 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
2020-08-21 19:04:24.399+0000 INFO  Note that since you did not explicitly set the port in dbms.connector.bolt.advertised_address Neo4j automatically set it to 11002 to match dbms.connector.bolt.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO  Note that since you did not explicitly set the port in dbms.connector.http.advertised_address Neo4j automatically set it to 11003 to match dbms.connector.http.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO  Starting...
2020-08-21 19:04:26.747+0000 INFO  ======== Neo4j 4.1.0 ========
Directories in use:
  home:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0
  config:       /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/conf
  logs:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/logs
  plugins:      /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/plugins
  import:       /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/import
  data:         /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data
  certificates: /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/certificates
  run:          /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/run
Starting Neo4j.
WARNING: Max 8192 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
2020-08-21 19:04:24.399+0000 INFO  Note that since you did not explicitly set the port in dbms.connector.bolt.advertised_address Neo4j automatically set it to 11002 to match dbms.connector.bolt.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO  Note that since you did not explicitly set the port in dbms.connector.http.advertised_address Neo4j automatically set it to 11003 to match dbms.connector.http.listen_address. This behavior may change in the future and we recommend you to explicitly set it.
2020-08-21 19:04:24.402+0000 INFO  Starting...
2020-08-21 19:04:26.747+0000 INFO  ======== Neo4j 4.1.0 ========
2020-08-21 19:04:32.005+0000 ERROR Failed to start Neo4j on dbms.connector.http.listen_address, a socket address. If missing port or hostname it is acquired from dbms.default_listen_address. Error starting Neo4j database server at /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data/databases
java.lang.RuntimeException: Error starting Neo4j database server at /home/edwardweymouth/.config/Neo4j Desktop/Application/neo4jDatabases/database-8c955e24-f305-4cd8-9342-490ffd3dc0ab/installation-4.1.0/data/databases
    at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198)
    at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:158)
    at com.neo4j.server.enterprise.EnterpriseManagementServiceFactory.createManagementService(EnterpriseManagementServiceFactory.java:38)
    at com.neo4j.server.enterprise.EnterpriseBootstrapper.createNeo(EnterpriseBootstrapper.java:20)
    at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:117)
    at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:87)
    at com.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:25)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'Server[backup-server]' was successfully initialized, but failed to start. Please see the attached cause exception "bind(..) failed: Address already in use".
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:463)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110)
    at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:189)
    ... 6 more
Caused by: io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
2020-08-21 19:04:32.006+0000 INFO  Neo4j Server shutdown initiated by request

debug.log

2020-08-21 19:04:24.897+0000 INFO [o.n.i.d.DiagnosticsManager] ********************************************************************************
2020-08-21 19:04:24.898+0000 INFO [o.n.i.d.DiagnosticsManager]                              [ System diagnostics ]                             
2020-08-21 19:04:24.899+0000 INFO [o.n.i.d.DiagnosticsManager] **************************

Answer 1

There was a lot of address already in use going on here, so it was kind of hard to parse, but this bit in the debug log turned out to be the important part:

2020-08-21 19:04:25.482+0000 INFO [c.n.c.c.TransactionBackupServiceProvider] Binding backup service on address 127.0.0.1:6362
2020-08-21 19:04:26.747+0000 INFO [c.n.s.e.EnterpriseNeoWebServer] ======== Neo4j 4.1.0 ========
2020-08-21 19:04:26.824+0000 INFO [o.n.b.BoltServer] Bolt server loaded
2020-08-21 19:04:26.930+0000 ERROR [c.n.c.n.Server] backup-server: cannot bind to '127.0.0.1:6362' with transport 'EpollServerSocketChannel'. bind(..) failed: Address already in use

searched for the port:

$ lsof -i :6362
COMMAND    PID           USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
java    530156 user  427u  IPv6 3670839      0t0  TCP localhost:6362 (LISTEN)
$ kill 530156

Looks like desktop leaves some java runnin' out in the fields there...

After that DB startup was all good.