On May 14, 2020, at 19:59, James <james@zuelow.net> wrote:

Hello all -

I've been dealing with a bit of a mystery for the past couple of days.

I run IRLP node 7295 using a 3.0 board mounted in an HP desktop, running Debian 10. The node feeds an NHRC-7 controller, which is in turn connected to a Kenwood TKR-750.

This setup has been extremely stable.

There have been some changes over the years, with the last hardware change being in 2019 when the old computer I was using started to fail, so I switched to a new HP.

Additional software is only OpenVPN as a secondary control link so I can disable the node from my phone if need be.

I have verified my router's port forwarding with nmap from my work, and I can see the open TCP ports, and when connected to the echo reflector I can also see the open UDP ports. My OpenVPN connection is still working as well, so I don't think there is a problem with the router or internal network.

scripts/audiotest works fine

I emulated that same test with `bin/key; play /usr/bin/soundtest.wav; bin/unkey` and that works. I didn't really need to do that, I was just at a point where I wanted something to work the way I thought it would. :)

So I'm confident that the IRLP node is sending audio to the NHRC-7, which is sending audio to the Kenwood, which is transmitting it.

scripts/audio_level_test also works, so I believe that the proper microphone is selected and the volume is correct. (Which means the Kenwood should be sending audio to the NHRC-7, which is sending it to the node right?) I get no clipping and an audio level of 69/100.

So everything SHOULD be working as far as I can tell. I've even done a re-install of the IRLP node from backup so I should have the latest & greatest IRLP files.


But I get no audio from IRLP. I can connect to the echo reflector and transmit to the node, but watching my router I don't see traffic other than the initial data connection. And I don't get any hint of a reply back from the echo reflector.


Any ideas? What am I missing?


Thanks!

James
KL2ZZ

I think there is a problem with your router. When I attempt to connect to your node on port 15425, it works, but it gives me the wrong IP address

repeater@tigger:~$ telnet stn7295 15425
Trying 24.237.7.242...
Connected to stn7295.
Escape character is '^]'.
stn7295 - kl2zz : Welcome 192.168.2.1

That is incorrect. It should say: Welcome 75.48.51.3

Which is my public IP. Something is causing external connections to NAT’d to your router internal IP.

-k9dc

On May 14, 2020, at 19:59, James <james@zuelow.net> wrote:

Hello all -

I've been dealing with a bit of a mystery for the past couple of days.

I run IRLP node 7295 using a 3.0 board mounted in an HP desktop, running Debian 10. The node feeds an NHRC-7 controller, which is in turn connected to a Kenwood TKR-750.

This setup has been extremely stable.

There have been some changes over the years, with the last hardware change being in 2019 when the old computer I was using started to fail, so I switched to a new HP.

Additional software is only OpenVPN as a secondary control link so I can disable the node from my phone if need be.

I have verified my router's port forwarding with nmap from my work, and I can see the open TCP ports, and when connected to the echo reflector I can also see the open UDP ports. My OpenVPN connection is still working as well, so I don't think there is a problem with the router or internal network.

scripts/audiotest works fine

I emulated that same test with `bin/key; play /usr/bin/soundtest.wav; bin/unkey` and that works. I didn't really need to do that, I was just at a point where I wanted something to work the way I thought it would. :)

So I'm confident that the IRLP node is sending audio to the NHRC-7, which is sending audio to the Kenwood, which is transmitting it.

scripts/audio_level_test also works, so I believe that the proper microphone is selected and the volume is correct. (Which means the Kenwood should be sending audio to the NHRC-7, which is sending it to the node right?) I get no clipping and an audio level of 69/100.

So everything SHOULD be working as far as I can tell. I've even done a re-install of the IRLP node from backup so I should have the latest & greatest IRLP files.


But I get no audio from IRLP. I can connect to the echo reflector and transmit to the node, but watching my router I don't see traffic other than the initial data connection. And I don't get any hint of a reply back from the echo reflector.


Any ideas? What am I missing?


Thanks!

James
KL2ZZ

Hmm.  OK.
Strange that OpenVPN works though.  The port forwards are the same, and it shouldn't have made any changes.
I'll play around with the router.
Thanks!
James
KL2ZZ
On 5/14/20 4:34 PM, k9dc wrote:

I think there is a problem with your router.  When I attempt to connect to your node on port 15425, it works, but it gives me the wrong IP address

repeater@tigger:~$ telnet stn7295 15425
Trying 24.237.7.242...
Connected to stn7295.
Escape character is '^]'.
stn7295 - kl2zz : Welcome 192.168.2.1

That is incorrect.  It should say: Welcome 75.48.51.3

Which is my public IP.  Something is causing external connections to NAT’d to your router internal IP.

-k9dc

On May 14, 2020, at 19:59, James <james@zuelow.net> wrote:

Hello all -

I've been dealing with a bit of a mystery for the past couple of days.

I run IRLP node 7295 using a 3.0 board mounted in an HP desktop, running Debian 10.  The node feeds an NHRC-7 controller, which is in turn connected to a Kenwood TKR-750.

This setup has been extremely stable.

There have been some changes over the years, with the last hardware change being in 2019 when the old computer I was using started to fail, so I switched to a new HP.

Additional software is only OpenVPN as a secondary control link so I can disable the node from my phone if need be.

I have verified my router's port forwarding with nmap from my work, and I can see the open TCP ports, and when connected to the echo reflector I can also see the open UDP ports.  My OpenVPN connection is still working as well, so I don't think there is a problem with the router or internal network.

scripts/audiotest works fine

I emulated that same test with `bin/key; play /usr/bin/soundtest.wav; bin/unkey` and that works.  I didn't really need to do that, I was just at a point where I wanted something to work the way I thought it would.  :)

So I'm confident that the IRLP node is sending audio to the NHRC-7, which is sending audio to the Kenwood, which is transmitting it.

scripts/audio_level_test also works, so I believe that the proper microphone is selected and the volume is correct.  (Which means the Kenwood should be sending audio to the NHRC-7, which is sending it to the node right?)  I get no clipping and an audio level of 69/100.

So everything SHOULD be working as far as I can tell.  I've even done a re-install of the IRLP node from backup so I should have the latest & greatest IRLP files.


But I get no audio from IRLP.  I can connect to the echo reflector and transmit to the node, but watching my router I don't see traffic other than the initial data connection.  And I don't get any hint of a reply back from the echo reflector.


Any ideas?  What am I missing?


Thanks!

James
KL2ZZ

A debian upgrade would not have changed the way your network routes, so something else has changed.

Do you have two routers?

You mention OpenVPN - how does it play into this? You need to have one or the other - port forwarding or VPN, not both.

Dave Cameron
VE7LTD

What is the IP of the node behind the router? What is 192.168.2.1?

Dave Cameron

On May 15, 2020, at 00:03, James <james@zuelow.net> wrote:

192.168.2.1 is the OpenWRT router.

192.168.2.10 is the node.


I tried turning off NAT on the OpenWRT router, and internet doesn't work at all. So I think it is designed to do that.

I did the telnet test, and I get reliable 2-way communication. (Well, sort of. I don't know how to interact beyond prompting the node to challenge me and then disconnect.)

And as I mentioned, the troubleshoot_irlp script reports success. It says that inbound TCP and UDP are good.


I'm not sure the NAT issue would have come up at exactly the same time as the node was updated. Perhaps I'll download the IRLP Debian 10 image and start over, but it is frustrating as everything seems to work.


On 5/14/20 7:38 PM, David Cameron - IRLP wrote:

What is the IP of the node behind the router? What is 192.168.2.1?

Dave Cameron

Your router is acting as some sort of proxy.

Rebuilding won't change this.

Dave

OK. I'll install a new router.  I'm pretty sure I have an old Linksys around with commercial firmware in it.

Amazing coincidence that the upgrade and router malfunction occurred at the same time.  That's puzzling to me.  But if it's the router, it's the router.


On 5/14/20 8:30 PM, David Cameron - IRLP wrote:

Your router is acting as some sort of proxy.

Rebuilding won't change this.

Dave

On May 15, 2020, at 14:04, James <james@zuelow.net> wrote:

You should be able to telnet to TCP 15425 and get your correct external IP in the response header. Hopefully we can rule out the router at this point.


I'm back to my original question:

* scripts/troubleshoot-irlp runs successfully. (See below)

* scripts/audio_level_test runs successfully, so I believe I have the right microphone selected in the sound system, and audio from the Kenwood is reaching the node computer.

* I can hear the local files (connected to, or files I play with key/unkey) so I know that the node is sending audio out to the Kenwood.


But if I connect to a reflector, there is no audio. Further, I can watch network traffic with iptables and there are no UDP packets arriving at the node. (But I can see the UDP packets arriving when I run the troubleshoot-irlp script.) There are some TCP packets coming in when I connect/disconnect. And the IRLP status page reflects these changes immediately, so I assume that's part of the traffic.

So it is like the node says it is connecting, but is not actually connecting.


Thanks!

James
KL2ZZ

Yes the new router NAT appears to be working now. I see my IP anyway. I do not understand how you are using iptables to check for traffic though. I would use a tool more like tcpdump (CLI version of wireshark).

Did you try the echo reflector (9990-9999) ?

-k9dc

On May 15, 2020, at 21:34, James <james@zuelow.net> wrote:

Oh I'm too lazy to tcpdump or even wireshark. I used to use tools like that but I over the years I find I rarely need to actually look at a packet. I am usually more interested in where a packet is going and why. I find I can get almost all of the information I need just by something like:

iptables -A INPUT -i eth0 -j LOG --log-prefix "IRLP: " (This one is a shotgun, it will log every packet arriving at eth0.)

Then I follow up with a tail -f /var/log/syslog | grep IRLP | grep -v ip.address.of.my.ssh.client | (other greps and awks as appropriate)

When I say I can see packets, I see them in real time as they get logged. Two terminals is great - one to run the script, one to watch the packets as they arrive.

If I need to go back and look at the logs, they're still there until logrotate clears them out.


Two mysteries:


1) why can I play local WAVs, but I can't hear IRLP audio?

2) why can I see the node computer send and receive traffic when I do the test, but not when I connect to an actual node?


This is why I'm about ready to wipe the disk, install new, and restore settings from backup. There is something seriously wrong with the node computer and it is acting inconsistently.

On 5/15/20 11:44 AM, k9dc wrote:
Yes the new router NAT appears to be working now. I see my IP anyway. I do not understand how you are using iptables to check for traffic though. I would use a tool more like tcpdump (CLI version of wireshark).

Did you try the echo reflector (9990-9999) ?

-k9dc

In that case, suit yourself. It’s a waste of time, but then, what do I know. Good luck
-k9dc