So I've made myself a simple git server using a raspberry Pi and a portable USB drive and it has been working well for some time. However, after a couple weeks of not having used it, I’m suddenly finding major problems.
Local system:
Server:
- Raspbian GNU/Linux 10 (buster)
I’m using SSH to access the server (which is headless). It is setup up very similar to the one here: https://pimylifeup.com/raspberry-pi-git-server/
Note: I’m not actually certain that the following symptoms are part of the same issue but suspect they are.
Issue Pt 1: Push, Pull and Clone Just Hang
The first thing I noticed was that pushing to my repo just hangs. Trying pull and clone had the same issue, as did cloning from a new, empty repo.
When I say “hang” here I mean that:
- At the terminal the command outputs “Cloning into ‘XYZ’…” or push/pull message and then sits with the little [] prompt. I’ve waited 30 minutes + and nothing happens.
- (When cloning) If I check the local file system at this time it has created the directory to clone into along with the .git directory, but none of the actual repo contents are copied.
- Eventually I've just used Ctrl-C to quit and return to the regular prompt. When cloning, this clears the directory it had made.
I’m just cloning like so:
git clone pi@192.168.2.40:/media/usb/REPONAME.git LOCAL_DIRECTORY
Where “pi” is what I’m calling my server, its IP is as you see, and /media/usb is where my portable drive is mounted.
Issue Pt 2: Some Commands Now Request Password
I was pretty convinced at first that I’d just mucked up the remote/upstream something or other, so among other things I tried these commands on my existing local repo (the one I was stuck on to start with, which FYI has a couple of commits I still need to push). “pi” is the name of my remote here.
git push --set-upstream pi master
or
git fetch pi
Either or these will prompt me for a password like so:
username@192.168.2.40's password:
But to my recollection I’ve configured no passwords for access to this git server nor do I particularly want them. And none of my go-to passwords (bad habit, yes) work.
Attempts to Resolve
- I’m able to SSH into the server just fine (no connectivity issues) and verified the external storage is still mounted and the files appear intact.
- I’ve run fsck on the USB drive with no problems reported.
- I’ve run git fsck --full on both server and local systems with no problems reported.
- I’ve rebooted both machines because.
- I’ve tried clearing and re-defining my remotes on the local system — maybe I mucked the config at some point? — but have the same results.
- I’ve tried initing a new repo on the server (worked fine) and cloning from that but with the same result.
Alas I’m pretty bad at git and Linux in general, but I have spent some hours Googling the issue to no avail.
Edit: SSH Echo Test Debug Log
Jasons-MacBook-Pro:~ jasonbond$ ssh -vvv pi@192.168.2.40 echo test
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.2.40 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.2.40 [192.168.2.40] port 22.
debug1: Connection established.
debug1: identity file /Users/jasonbond/.ssh/id_rsa type -1
debug1: identity file /Users/jasonbond/.ssh/id_rsa-cert type -1
debug1: identity file /Users/jasonbond/.ssh/id_dsa type -1
debug1: identity file /Users/jasonbond/.ssh/id_dsa-cert type -1
debug1: identity file /Users/jasonbond/.ssh/id_ecdsa type -1
debug1: identity file /Users/jasonbond/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/jasonbond/.ssh/id_ed25519 type -1
debug1: identity file /Users/jasonbond/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/jasonbond/.ssh/id_xmss type -1
debug1: identity file /Users/jasonbond/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Raspbian-10+deb10u2
debug1: match: OpenSSH_7.9p1 Raspbian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 192.168.2.40:22 as 'pi'
debug3: hostkeys_foreach: reading file "/Users/jasonbond/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/jasonbond/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from 192.168.2.40
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:Ny2qJQSyutMKEw6CJPdtK0YLRlkqUiOikUobseVhtSY
debug3: hostkeys_foreach: reading file "/Users/jasonbond/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/jasonbond/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from 192.168.2.40
debug1: Host '192.168.2.40' is known and matches the ECDSA host key.
debug1: Found key in /Users/jasonbond/.ssh/known_hosts:4
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/jasonbond/.ssh/id_rsa
debug1: Will attempt key: /Users/jasonbond/.ssh/id_dsa
debug1: Will attempt key: /Users/jasonbond/.ssh/id_ecdsa
debug1: Will attempt key: /Users/jasonbond/.ssh/id_ed25519
debug1: Will attempt key: /Users/jasonbond/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/jasonbond/.ssh/id_rsa
debug3: no such identity: /Users/jasonbond/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /Users/jasonbond/.ssh/id_dsa
debug3: no such identity: /Users/jasonbond/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /Users/jasonbond/.ssh/id_ecdsa
debug3: no such identity: /Users/jasonbond/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /Users/jasonbond/.ssh/id_ed25519
debug3: sign_and_send_pubkey: ED25519 SHA256:7HRj7G4gFYou5AEvDr6uExmKNSbvaLnFSIUM4ymKRr8
debug3: sign_and_send_pubkey: signing using ssh-ed25519
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.2.40 ([192.168.2.40]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 4
debug1: Remote:
