Topics

Debian upgrade breaks IRLP?

James <james@...>
 

Oh I'm sorry.  Was there some advice on steps to take with my node that I missed?

I didn't recall getting any - I will go back through the online archives to see if there was mail I didn't get.


Thanks for all the help guys!


73,


James

KL2ZZ

On 5/15/20 6:10 PM, k9dc wrote:
In that case, suit yourself. It’s a waste of time, but then, what do I know. Good luck
-k9dc

k9dc
 

In that case, suit yourself. It’s a waste of time, but then, what do I know. Good luck
-k9dc

On May 15, 2020, at 21:34, James <@kl2zz> wrote:

Oh I'm too lazy to tcpdump or even wireshark. I used to use tools like that but I over the years I find I rarely need to actually look at a packet. I am usually more interested in where a packet is going and why. I find I can get almost all of the information I need just by something like:

iptables -A INPUT -i eth0 -j LOG --log-prefix "IRLP: " (This one is a shotgun, it will log every packet arriving at eth0.)

Then I follow up with a tail -f /var/log/syslog | grep IRLP | grep -v ip.address.of.my.ssh.client | (other greps and awks as appropriate)

When I say I can see packets, I see them in real time as they get logged. Two terminals is great - one to run the script, one to watch the packets as they arrive.

If I need to go back and look at the logs, they're still there until logrotate clears them out.


Two mysteries:


1) why can I play local WAVs, but I can't hear IRLP audio?

2) why can I see the node computer send and receive traffic when I do the test, but not when I connect to an actual node?


This is why I'm about ready to wipe the disk, install new, and restore settings from backup. There is something seriously wrong with the node computer and it is acting inconsistently.


On 5/15/20 11:44 AM, k9dc wrote:
Yes the new router NAT appears to be working now. I see my IP anyway. I do not understand how you are using iptables to check for traffic though. I would use a tool more like tcpdump (CLI version of wireshark).

Did you try the echo reflector (9990-9999) ?

-k9dc

James <james@...>
 

Oh I'm too lazy to tcpdump or even wireshark.  I used to use tools like that but I over the years I find I rarely need to actually look at a packet.  I am usually more interested in where a packet is going and why.  I find I can get almost all of the information I need just by something like:

iptables -A INPUT -i eth0 -j LOG --log-prefix "IRLP: "  (This one is a shotgun, it will log every packet arriving at eth0.)

Then I follow up with a tail -f /var/log/syslog | grep IRLP | grep -v ip.address.of.my.ssh.client | (other greps and awks as appropriate)

When I say I can see packets, I see them in real time as they get logged.  Two terminals is great - one to run the script, one to watch the packets as they arrive.

If I need to go back and look at the logs, they're still there until logrotate clears them out.


Two mysteries:


1) why can I play local WAVs, but I can't hear IRLP audio?

2) why can I see the node computer send and receive traffic when I do the test, but not when I connect to an actual node?


This is why I'm about ready to wipe the disk, install new, and restore settings from backup.  There is something seriously wrong with the node computer and it is acting inconsistently.

On 5/15/20 11:44 AM, k9dc wrote:
Yes the new router NAT appears to be working now. I see my IP anyway. I do not understand how you are using iptables to check for traffic though. I would use a tool more like tcpdump (CLI version of wireshark).

Did you try the echo reflector (9990-9999) ?

-k9dc

k9dc
 

Yes the new router NAT appears to be working now. I see my IP anyway. I do not understand how you are using iptables to check for traffic though. I would use a tool more like tcpdump (CLI version of wireshark).

Did you try the echo reflector (9990-9999) ?

-k9dc

On May 15, 2020, at 14:04, James <@kl2zz> wrote:

You should be able to telnet to TCP 15425 and get your correct external IP in the response header. Hopefully we can rule out the router at this point.


I'm back to my original question:

* scripts/troubleshoot-irlp runs successfully. (See below)

* scripts/audio_level_test runs successfully, so I believe I have the right microphone selected in the sound system, and audio from the Kenwood is reaching the node computer.

* I can hear the local files (connected to, or files I play with key/unkey) so I know that the node is sending audio out to the Kenwood.


But if I connect to a reflector, there is no audio. Further, I can watch network traffic with iptables and there are no UDP packets arriving at the node. (But I can see the UDP packets arriving when I run the troubleshoot-irlp script.) There are some TCP packets coming in when I connect/disconnect. And the IRLP status page reflects these changes immediately, so I assume that's part of the traffic.

So it is like the node says it is connecting, but is not actually connecting.


Thanks!

James
KL2ZZ

James <james@...>
 

You should be able to telnet to TCP 15425 and get your correct external IP in the response header.  Hopefully we can rule out the router at this point.


I'm back to my original question:

* scripts/troubleshoot-irlp runs successfully.  (See below)

* scripts/audio_level_test runs successfully, so I believe I have the right microphone selected in the sound system, and audio from the Kenwood is reaching the node computer.

* I can hear the local files (connected to, or files I play with key/unkey) so I know that the node is sending audio out to the Kenwood.


But if I connect to a reflector, there is no audio.  Further, I can watch network traffic with iptables and there are no UDP packets arriving at the node.  (But I can see the UDP packets arriving when I run the troubleshoot-irlp script.)  There are some TCP packets coming in when I connect/disconnect.  And the IRLP status page reflects these changes immediately, so I assume that's part of the traffic.

So it is like the node says it is connecting, but is not actually connecting.


Thanks!

James
KL2ZZ


-----


**********************************
*TCP and UDP Port Forwarding Test*
**********************************

Performing INBOUND UDP and TCP Port Forwarding Test
Detecting Incoming IP = 24.237.7.242
Testing TCP and UDP ports .. done.

-----------------------------------------------------------------------------
TEST No. 2a REPORT - TCP PASS - TCP port 15425 is forwarded correctly.
Waiting 4 seconds .... done.
TEST No. 2b REPORT - UDP PASS - UDP ports 2074-2093 are forwarded correctly.
TEST No. 2c REPORT - EchoLink UDP ERROR - The following UDP port(s):
 5198  5199  is/are not forwarded correctly.
-----------------------------------------------------------------------------

Performing OUTBOUND Port Test
Testing OUTBOUND TCP and UDP ports ........................... done.

------------------------------------------------------------------------------
TEST No. 3 REPORT - PASS - All the ports are working OUTBOUND correctly
------------------------------------------------------------------------------

Press ENTER to continue :

On 5/14/20 9:01 PM, James wrote:
OK. I'll install a new router.  I'm pretty sure I have an old Linksys around with commercial firmware in it.

Amazing coincidence that the upgrade and router malfunction occurred at the same time.  That's puzzling to me.  But if it's the router, it's the router.


On 5/14/20 8:30 PM, David Cameron - IRLP wrote:
Your router is acting as some sort of proxy.

Rebuilding won't change this.

Dave

James <james@...>
 

OK.  I'll install a new router.  I'm pretty sure I have an old Linksys around with commercial firmware in it.

Amazing coincidence that the upgrade and router malfunction occurred at the same time.  That's puzzling to me.  But if it's the router, it's the router.

On 5/14/20 8:30 PM, David Cameron - IRLP wrote:
Your router is acting as some sort of proxy.

Rebuilding won't change this.

Dave

David Cameron - IRLP
 

Your router is acting as some sort of proxy. 

Rebuilding won't change this. 

Dave

-------- Original message --------
From: k9dc <Dave@...>
Date: 5/14/20 9:16 PM (GMT-08:00)
To: IRLP@irlp.groups.io
Subject: Re: [IRLP] Debian upgrade breaks IRLP?


Your router needs to NAT outbound to your public IP 24.237.7.242.  But it should leave inbound connections alone and not NAT to your router address of 192.168.2.1. That appears to be what it is doing.  Sorry I do not use OpenWRT here, so I cannot help much beyond that.

Using IRLP VPN would also probably fix it, but that should not be necessary. As DaveC said, nothing about upgrading Debian would affect your routing.

-k9dc


> On May 15, 2020, at 00:03, James <james@...> wrote:
>
> 192.168.2.1 is the OpenWRT router.
>
> 192.168.2.10 is the node.
>
>
> I tried turning off NAT on the OpenWRT router, and internet doesn't work at all.  So I think it is designed to do that.
>
> I did the telnet test, and I get reliable 2-way communication. (Well, sort of.  I don't know how to interact beyond prompting the node to challenge me and then disconnect.)
>
> And as I mentioned, the troubleshoot_irlp script reports success. It says that inbound TCP and UDP are good.
>
>
> I'm not sure the NAT issue would have come up at exactly the same time as the node was updated.  Perhaps I'll download the IRLP Debian 10 image and start over, but it is frustrating as everything seems to work.
>
>
> On 5/14/20 7:38 PM, David Cameron - IRLP wrote:
>> What is the IP of the node behind the router? What is 192.168.2.1?
>>
>> Dave Cameron
>>
>>





k9dc
 

Your router needs to NAT outbound to your public IP 24.237.7.242. But it should leave inbound connections alone and not NAT to your router address of 192.168.2.1. That appears to be what it is doing. Sorry I do not use OpenWRT here, so I cannot help much beyond that.

Using IRLP VPN would also probably fix it, but that should not be necessary. As DaveC said, nothing about upgrading Debian would affect your routing.

-k9dc

On May 15, 2020, at 00:03, James <@kl2zz> wrote:

192.168.2.1 is the OpenWRT router.

192.168.2.10 is the node.


I tried turning off NAT on the OpenWRT router, and internet doesn't work at all. So I think it is designed to do that.

I did the telnet test, and I get reliable 2-way communication. (Well, sort of. I don't know how to interact beyond prompting the node to challenge me and then disconnect.)

And as I mentioned, the troubleshoot_irlp script reports success. It says that inbound TCP and UDP are good.


I'm not sure the NAT issue would have come up at exactly the same time as the node was updated. Perhaps I'll download the IRLP Debian 10 image and start over, but it is frustrating as everything seems to work.


On 5/14/20 7:38 PM, David Cameron - IRLP wrote:
What is the IP of the node behind the router? What is 192.168.2.1?

Dave Cameron

James <james@...>
 

192.168.2.1 is the OpenWRT router.

192.168.2.10 is the node.


I tried turning off NAT on the OpenWRT router, and internet doesn't work at all.  So I think it is designed to do that.

I did the telnet test, and I get reliable 2-way communication. (Well, sort of.  I don't know how to interact beyond prompting the node to challenge me and then disconnect.)

And as I mentioned, the troubleshoot_irlp script reports success. It says that inbound TCP and UDP are good.


I'm not sure the NAT issue would have come up at exactly the same time as the node was updated.  Perhaps I'll download the IRLP Debian 10 image and start over, but it is frustrating as everything seems to work.

On 5/14/20 7:38 PM, David Cameron - IRLP wrote:
What is the IP of the node behind the router? What is 192.168.2.1?

Dave Cameron

David Cameron - IRLP
 

What is the IP of the node behind the router? What is 192.168.2.1?

Dave Cameron 

-------- Original message --------
From: James <james@...>
Date: 5/14/20 8:01 PM (GMT-08:00)
To: irlp@groups.io
Subject: Re: [IRLP] Debian upgrade breaks IRLP?

I only have one router, running OpenWRT.  I use OpenVPN as a standby
control circuit - it isn't involved in IRLP at all, except that it
allows me to control the node from my phone.  The OpenVPN server is
actually running on the node, and the only OpenVPN client is my phone. 
The router port forwards UDP 1194 through to the node computer.

I was wondering about the upgrade, as the outage coincided with the
reboot after the upgrade.

James
KL2ZZ


On 5/14/20 5:40 PM, David Cameron - IRLP wrote:
> A debian upgrade would not have changed the way your network routes,
> so something else has changed.
>
> Do you have two routers?
>
> You mention OpenVPN - how does it play into this? You need to have one
> or the other - port forwarding or VPN, not both.
>
> Dave Cameron
> VE7LTD
>
>


James <james@...>
 

I only have one router, running OpenWRT.  I use OpenVPN as a standby control circuit - it isn't involved in IRLP at all, except that it allows me to control the node from my phone.  The OpenVPN server is actually running on the node, and the only OpenVPN client is my phone.  The router port forwards UDP 1194 through to the node computer.

I was wondering about the upgrade, as the outage coincided with the reboot after the upgrade.

James
KL2ZZ

On 5/14/20 5:40 PM, David Cameron - IRLP wrote:
A debian upgrade would not have changed the way your network routes, so something else has changed.

Do you have two routers?

You mention OpenVPN - how does it play into this? You need to have one or the other - port forwarding or VPN, not both.

Dave Cameron
VE7LTD

David Cameron - IRLP
 

A debian upgrade would not have changed the way your network routes, so something else has changed.

Do you have two routers?

You mention OpenVPN - how does it play into this? You need to have one or the other - port forwarding or VPN, not both.

Dave Cameron
VE7LTD

On 2020-05-14 6:33 p.m., James wrote:
Hmm.  OK.
Strange that OpenVPN works though.  The port forwards are the same, and it shouldn't have made any changes.
I'll play around with the router.
Thanks!
James
KL2ZZ
On 5/14/20 4:34 PM, k9dc wrote:
I think there is a problem with your router.  When I attempt to connect to your node on port 15425, it works, but it gives me the wrong IP address

repeater@tigger:~$ telnet stn7295 15425
Trying 24.237.7.242...
Connected to stn7295.
Escape character is '^]'.
stn7295 - kl2zz : Welcome 192.168.2.1

That is incorrect.  It should say: Welcome 75.48.51.3

Which is my public IP.  Something is causing external connections to NAT’d to your router internal IP.

-k9dc


On May 14, 2020, at 19:59, James <@kl2zz> wrote:

Hello all -

I've been dealing with a bit of a mystery for the past couple of days.

I run IRLP node 7295 using a 3.0 board mounted in an HP desktop, running Debian 10.  The node feeds an NHRC-7 controller, which is in turn connected to a Kenwood TKR-750.

This setup has been extremely stable.

There have been some changes over the years, with the last hardware change being in 2019 when the old computer I was using started to fail, so I switched to a new HP.

Additional software is only OpenVPN as a secondary control link so I can disable the node from my phone if need be.

I have verified my router's port forwarding with nmap from my work, and I can see the open TCP ports, and when connected to the echo reflector I can also see the open UDP ports.  My OpenVPN connection is still working as well, so I don't think there is a problem with the router or internal network.

scripts/audiotest works fine

I emulated that same test with `bin/key; play /usr/bin/soundtest.wav; bin/unkey` and that works.  I didn't really need to do that, I was just at a point where I wanted something to work the way I thought it would.  :)

So I'm confident that the IRLP node is sending audio to the NHRC-7, which is sending audio to the Kenwood, which is transmitting it.

scripts/audio_level_test also works, so I believe that the proper microphone is selected and the volume is correct.  (Which means the Kenwood should be sending audio to the NHRC-7, which is sending it to the node right?)  I get no clipping and an audio level of 69/100.

So everything SHOULD be working as far as I can tell.  I've even done a re-install of the IRLP node from backup so I should have the latest & greatest IRLP files.


But I get no audio from IRLP.  I can connect to the echo reflector and transmit to the node, but watching my router I don't see traffic other than the initial data connection.  And I don't get any hint of a reply back from the echo reflector.


Any ideas?  What am I missing?


Thanks!

James
KL2ZZ

James <james@...>
 

Hmm.  OK.

Strange that OpenVPN works though.  The port forwards are the same, and it shouldn't have made any changes.


I'll play around with the router.


Thanks!


James
KL2ZZ

On 5/14/20 4:34 PM, k9dc wrote:
I think there is a problem with your router. When I attempt to connect to your node on port 15425, it works, but it gives me the wrong IP address

repeater@tigger:~$ telnet stn7295 15425
Trying 24.237.7.242...
Connected to stn7295.
Escape character is '^]'.
stn7295 - kl2zz : Welcome 192.168.2.1

That is incorrect. It should say: Welcome 75.48.51.3

Which is my public IP. Something is causing external connections to NAT’d to your router internal IP.

-k9dc


On May 14, 2020, at 19:59, James <@kl2zz> wrote:

Hello all -

I've been dealing with a bit of a mystery for the past couple of days.

I run IRLP node 7295 using a 3.0 board mounted in an HP desktop, running Debian 10. The node feeds an NHRC-7 controller, which is in turn connected to a Kenwood TKR-750.

This setup has been extremely stable.

There have been some changes over the years, with the last hardware change being in 2019 when the old computer I was using started to fail, so I switched to a new HP.

Additional software is only OpenVPN as a secondary control link so I can disable the node from my phone if need be.

I have verified my router's port forwarding with nmap from my work, and I can see the open TCP ports, and when connected to the echo reflector I can also see the open UDP ports. My OpenVPN connection is still working as well, so I don't think there is a problem with the router or internal network.

scripts/audiotest works fine

I emulated that same test with `bin/key; play /usr/bin/soundtest.wav; bin/unkey` and that works. I didn't really need to do that, I was just at a point where I wanted something to work the way I thought it would. :)

So I'm confident that the IRLP node is sending audio to the NHRC-7, which is sending audio to the Kenwood, which is transmitting it.

scripts/audio_level_test also works, so I believe that the proper microphone is selected and the volume is correct. (Which means the Kenwood should be sending audio to the NHRC-7, which is sending it to the node right?) I get no clipping and an audio level of 69/100.

So everything SHOULD be working as far as I can tell. I've even done a re-install of the IRLP node from backup so I should have the latest & greatest IRLP files.


But I get no audio from IRLP. I can connect to the echo reflector and transmit to the node, but watching my router I don't see traffic other than the initial data connection. And I don't get any hint of a reply back from the echo reflector.


Any ideas? What am I missing?


Thanks!

James
KL2ZZ

k9dc
 

I think there is a problem with your router. When I attempt to connect to your node on port 15425, it works, but it gives me the wrong IP address

repeater@tigger:~$ telnet stn7295 15425
Trying 24.237.7.242...
Connected to stn7295.
Escape character is '^]'.
stn7295 - kl2zz : Welcome 192.168.2.1

That is incorrect. It should say: Welcome 75.48.51.3

Which is my public IP. Something is causing external connections to NAT’d to your router internal IP.

-k9dc

On May 14, 2020, at 19:59, James <@kl2zz> wrote:

Hello all -

I've been dealing with a bit of a mystery for the past couple of days.

I run IRLP node 7295 using a 3.0 board mounted in an HP desktop, running Debian 10. The node feeds an NHRC-7 controller, which is in turn connected to a Kenwood TKR-750.

This setup has been extremely stable.

There have been some changes over the years, with the last hardware change being in 2019 when the old computer I was using started to fail, so I switched to a new HP.

Additional software is only OpenVPN as a secondary control link so I can disable the node from my phone if need be.

I have verified my router's port forwarding with nmap from my work, and I can see the open TCP ports, and when connected to the echo reflector I can also see the open UDP ports. My OpenVPN connection is still working as well, so I don't think there is a problem with the router or internal network.

scripts/audiotest works fine

I emulated that same test with `bin/key; play /usr/bin/soundtest.wav; bin/unkey` and that works. I didn't really need to do that, I was just at a point where I wanted something to work the way I thought it would. :)

So I'm confident that the IRLP node is sending audio to the NHRC-7, which is sending audio to the Kenwood, which is transmitting it.

scripts/audio_level_test also works, so I believe that the proper microphone is selected and the volume is correct. (Which means the Kenwood should be sending audio to the NHRC-7, which is sending it to the node right?) I get no clipping and an audio level of 69/100.

So everything SHOULD be working as far as I can tell. I've even done a re-install of the IRLP node from backup so I should have the latest & greatest IRLP files.


But I get no audio from IRLP. I can connect to the echo reflector and transmit to the node, but watching my router I don't see traffic other than the initial data connection. And I don't get any hint of a reply back from the echo reflector.


Any ideas? What am I missing?


Thanks!

James
KL2ZZ

James <james@...>
 

Hello all -

I've been dealing with a bit of a mystery for the past couple of days.

I run IRLP node 7295 using a 3.0 board mounted in an HP desktop, running Debian 10.  The node feeds an NHRC-7 controller, which is in turn connected to a Kenwood TKR-750.

This setup has been extremely stable.

There have been some changes over the years, with the last hardware change being in 2019 when the old computer I was using started to fail, so I switched to a new HP.

Additional software is only OpenVPN as a secondary control link so I can disable the node from my phone if need be.

I generally stay on top of OS updates.  I performed a full-upgrade on the 12th, after Debian released the last point release for Buster over the weekend.  During the upgrade I was frustrated that tab completion wasn't working in my console so I added bash-completion.  That update upgraded the following packages:


Installed:
bash-completion:all 1:2.8-6
linux-image-4.19.0-9-686-pae:i386 4.19.118-2
linux-image-4.19.0-9-686-pae:i386 4.19.118-2
linux-image-4.19.0-9-686-pae:i386 4.19.118-2
linux-image-4.19.0-9-686-pae:i386 4.19.118-2
Upgraded:
base-files:i386 10.3+deb10u3 => 10.3+deb10u4
iputils-ping:i386 3:20180629-2 => 3:20180629-2+deb10u1
libfuse2:i386 2.9.9-1 => 2.9.9-1+deb10u1
libnss-systemd:i386 241-7~deb10u3 => 241-7~deb10u4
libpam-systemd:i386 241-7~deb10u3 => 241-7~deb10u4
libsystemd0:i386 241-7~deb10u3 => 241-7~deb10u4
libudev1:i386 241-7~deb10u3 => 241-7~deb10u4
linux-image-686-pae:i386 4.19+105+deb10u3 => 4.19+105+deb10u4
linux-libc-dev:i386 4.19.98-1+deb10u1 => 4.19.118-2
postfix-sqlite:i386 3.4.8-0+10debu1 => 3.4.10-0+deb10u1
postfix:i386 3.4.8-0+10debu1 => 3.4.10-0+deb10u1
systemd-sysv:i386 241-7~deb10u3 => 241-7~deb10u4
systemd:i386 241-7~deb10u3 => 241-7~deb10u4
tzdata:all 2019c-0+deb10u1 => 2020a-0+deb10u1
udev:i386 241-7~deb10u3 => 241-7~deb10u4
wpasupplicant:i386 2:2.7+git20190128+0c1e29f-6+deb10u1 => 2:2.7+git20190128+0c1e29f-6+deb10u2
.
Removed:
linux-image-4.19.0-6-686-pae:i386 4.19.67-2+deb10u2
Purged:
linux-image-4.19.0-6-686-pae:i386 4.19.67-2+deb10u2

Since the upgrade, my node does not seem as if it sending or receiving IRLP audio from the Internet, although all tests I can do seem to be successful.

scripts/troubleshoot_irlp is successful (except for 2c since I'm not running EchoIRLP).

I have verified my router's port forwarding with nmap from my work, and I can see the open TCP ports, and when connected to the echo reflector I can also see the open UDP ports.  My OpenVPN connection is still working as well, so I don't think there is a problem with the router or internal network.

scripts/audiotest works fine

I emulated that same test with `bin/key; play /usr/bin/soundtest.wav; bin/unkey` and that works.  I didn't really need to do that, I was just at a point where I wanted something to work the way I thought it would.  :)

So I'm confident that the IRLP node is sending audio to the NHRC-7, which is sending audio to the Kenwood, which is transmitting it.

scripts/audio_level_test also works, so I believe that the proper microphone is selected and the volume is correct.  (Which means the Kenwood should be sending audio to the NHRC-7, which is sending it to the node right?)  I get no clipping and an audio level of 69/100.

So everything SHOULD be working as far as I can tell.  I've even done a re-install of the IRLP node from backup so I should have the latest & greatest IRLP files.


But I get no audio from IRLP.  I can connect to the echo reflector and transmit to the node, but watching my router I don't see traffic other than the initial data connection.  And I don't get any hint of a reply back from the echo reflector.


Any ideas?  What am I missing?


Thanks!

James
KL2ZZ