WAN-facing web server inaccessible from LAN

From: CHYRON (DSMITHHFX)31 Jan 18:16
To: ALL1 of 24
Made the ISP switch and got internet access to the web/file servers.

Can't hit the web server from our LAN, either by IP or domain.

I can ssh in, export and mount NFS shares locally. I can ping the 'reserved' LAN IP but not the web-facing static IP.

Tried disabling UFW and Fail2ban, no joy ... fast running out of ideas.
From: william (WILLIAMA)31 Jan 18:38
To: CHYRON (DSMITHHFX) 2 of 24
Can you see the server from outside the LAN? Any dodgy hosts files left over from dev/test?
From: CHYRON (DSMITHHFX)31 Jan 18:52
To: william (WILLIAMA) 3 of 24
Yeah I can see it from outside. Nothing weird in Hosts, /dev/tests ? - no such animal here.
From: CHYRON (DSMITHHFX)31 Jan 22:19
To: ALL4 of 24
New ISPs modem-router DHCP server seems flaky, shows random devices as connected/disconnected (when I know they're connected). This reared its head when my boss lost printing and I could not discover the printer on the network anymore.
From: william (WILLIAMA)31 Jan 23:21
To: CHYRON (DSMITHHFX) 5 of 24
I meant during development and testing. Relieved to hear you don't have truck with such nonsense.
From: william (WILLIAMA) 1 Feb 15:42
To: CHYRON (DSMITHHFX) 6 of 24
Have you fixed it yet? 

Oh yeah, meant to ask, from outside your LAN, can you see the web server both by public/WAN IP address and domain name?
EDITED: 1 Feb 15:52 by WILLIAMA
From: CHYRON (DSMITHHFX) 2 Feb 14:10
To: william (WILLIAMA) 7 of 24
LAN access to web server not fixed, I can hit it from outside by domain and ip.

I learned that the ISP has blocked port 22 so I changed the NAT to 30 outside --> 22 in and can now ssh in from outside (haven't figured out if/how that works with sftp yet).

I also learned the printer had the old 192.168.0.x ip hard-coded in. Fortunately our old ISP is still connected, so I was able to hook the printer and a pc up to that and webmin in to turn dhcp on, also applied latest firmware (ca. 2015) to it, to restore LAN printing.

Will be back in the office Friday for more fun.
From: william (WILLIAMA) 2 Feb 14:29
To: CHYRON (DSMITHHFX) 8 of 24
Yeah, I know there are a few ISPs who do that, although I'd have thought it was a bit odd with a business customer. 

Not being able to see your server from the LAN is a bit of a puzzle. If you can see it from outside the LAN with the IP address, then I can't really understand why you can't see it from the LAN. I mean, your query heads off to the ISP and after that you should, in theory, be in the same position as somebody on the outside.

What does traceroute show?
EDITED: 2 Feb 14:31 by WILLIAMA
From: CHYRON (DSMITHHFX) 2 Feb 14:51
To: william (WILLIAMA) 9 of 24
Yeah it's kind of weird. I suspect that Bell's 'small business' offering is identical to their residential service, but more expensive? Anyhoo I'm wondering if (based on the ssh thing) maybe they've reserved ports 80 and 443 for LAN access to the modem webmin*, and forgot to tell anybody (like they did with ssh). Documentation isn't their strong suit as it's all geared to non-techincal people doing non-technical things, no depth to it whatever. There are a few independent forums (e.g. reddit, dslreports) with a lot of stuff about these kinds of issues, slogging through them now.

*Edit: just remembered I can hit another web server (this one on an ancient powermac) on our LAN by IP.
 (pyfitg)
EDITED: 2 Feb 15:12 by DSMITHHFX
From: CHYRON (DSMITHHFX) 2 Feb 15:07
To: william (WILLIAMA) 10 of 24
Not used traceroute, I suppose it's one more thing I'll have to learn  :-@

There's the raw output. I tried default (30-hops) and -m 60 but it dies at 15 in both cases. (this also has the -I flag):
 
Code: 
[08:57 dave dave-fc32~]traceroute -I -m 60 xxxxxxxx.com
traceroute to xxxxxxx.com (xxx.xxx.xxx.xxx), 60 hops max, 60 byte packets
 1  _gateway (192.168.1.1)  0.695 ms  0.705 ms  0.852 ms
 2  dsl-173-206-32-1.tor.primus.ca (173.206.32.1)  13.157 ms  15.164 ms  17.275 ms
 3  10.201.117.14 (10.201.117.14)  19.925 ms  20.349 ms  22.580 ms
 4  10.201.117.73 (10.201.117.73)  24.113 ms  26.371 ms  27.482 ms
 5  10.201.121.49 (10.201.121.49)  29.515 ms  33.615 ms  34.703 ms
 6  64.125.60.212 (64.125.60.212)  36.427 ms  36.433 ms  38.614 ms
 7  * * ae4.mpr1.yyz1.ca.zip.zayo.com (64.125.23.123)  14.315 ms
 8  * * *
 9  * * *
10  bx10-newyork83_ae10.net.bell.ca (184.150.181.80)  35.782 ms  37.475 ms  38.262 ms
11  tcore4-newyorkaa_bundle-ether8.net.bell.ca (64.230.79.210)  59.575 ms  59.834 ms  59.827 ms
12  tcore3-montreal02_hundredgige1-12-0-0.net.bell.ca (64.230.79.149)  60.862 ms  52.664 ms  52.725 ms
13  tcore3-toronto21_1-6-0-0.net.bell.ca (64.230.79.30)  58.291 ms  45.906 ms  44.223 ms
14  tcore4-toronto01_0-1-0-0.net.bell.ca (64.230.50.234)  45.356 ms  39.585 ms  48.195 ms
15  lns1-toronto01_xe-7\0472\0470_ae1.net.bell.ca (64.230.102.35)  40.066 ms  40.258 ms  35.764 ms
EDITED: 3 Feb 15:20 by DSMITHHFX
From: CHYRON (DSMITHHFX) 2 Feb 15:38
To: ALL11 of 24
Here's some kind of explanation:
Quote: 
to my best of knowledge, the HH2000 doesn't support nat loopback (this is what you need to access your local server using the wan IP address. there's no way around this. you could have your home dhcp and dns server on your network... and override your domain IP address on your local network to point it to the local lan IP address.
More detail:

https://serverfault.com/questions/55611/loopback-to-forwarded-public-ip-address-from-local-network-hairpin-nat
From: william (WILLIAMA) 2 Feb 16:45
To: CHYRON (DSMITHHFX) 12 of 24
Doesn't look like the trace will help. You're getting Bell servers that won't respond to trace requests so it just times out. I can see your site and my tracert timed out too. Shame really because it's useful sometimes.

Interesting thing though, I plead ignorance seeing as it's at least 10 years since I played with any network stuff in anything remotely like a professional way. I had a quick squint at my owncloud server which is open to the internet. Turns out that the traceroute for that is totally different from my LAN to when I look at it from outside. From the LAN it goes to the router and from there straight back to the server, ie just two hops. Presumably the router is smart enough not to send a local request outside. But my router is a single (external) IP device for domestic use. Is the router you've been provided with handling a batch of external IP addresses? If so, does it make a difference whether you are on the same LAN as the web/file server, or whatever?
From: william (WILLIAMA) 2 Feb 16:55
To: william (WILLIAMA) 13 of 24
Aah, didn't see that last message until after mine.
From: CHYRON (DSMITHHFX) 2 Feb 17:04
To: william (WILLIAMA) 14 of 24
I've worked out a couple of things to try. Complicating this is an .htaccess redirect from 80-443 (http-https).

I've set up a noip.com dynamic dns domain pointing at the WAN iP (I suspect this will wind up in the same bad place though)

or

I can NFS export the server's /var/www/html and point a virtualhost on another server and so see the WAN-facing *content* which is the main thing (always nice to confirm if it can be hit from the actual domain though)

A third option to consider is monkeying with another .htaccess rule to override the redirect for LAN requests but I dunno

A fourth option is to rejig the managed switch for DHCP, port forward the modem to that, and port forward the server from it.

Not liking any but worth a try, maybe.

Someone helpfully pointed out that these modems are 'internet for idiots,' which serve the market very well so why would they change that? (our old ISPs modem has no such issues, but it's more business oriented; Bell is late to the party)
EDITED: 2 Feb 17:05 by DSMITHHFX
From: william (WILLIAMA) 2 Feb 17:04
To: CHYRON (DSMITHHFX) 15 of 24
That's sad. Is it going to be a problem for the business? 
From: CHYRON (DSMITHHFX) 2 Feb 17:08
To: william (WILLIAMA) 16 of 24
Not really. The main thing is hitting the server from outside for all the WFH folks, it does that np. Also I mostly WFH ...

Verifying sftp access is my next priority.
From: william (WILLIAMA) 2 Feb 17:21
To: CHYRON (DSMITHHFX) 17 of 24
Not 100% sure why the redirect to HTTPS is a problem (I have that on my server) but as I've contributed nothing useful so far, I shall leave you to continue enjoying yourself with network fun.
 
From: CHYRON (DSMITHHFX) 2 Feb 17:32
To: CHYRON (DSMITHHFX) 18 of 24
sftp over port 30 works.
From: patch 3 Feb 09:14
To: CHYRON (DSMITHHFX) 19 of 24
If you're trying to use a consumer grade router/modem/firewall thing for vaguely business purposes, then all bets are off, really. There's a reason people pay more for business grade stuff: it let's you do the things you need to do. Thing like NATting  out of the router's external address and hairpinning back in again to the outbound address which is then NATted to an internal server (which gives the SecOps side of my career the heebies, by the way).

But anyway, do you have a DNS server running on the LAN, maybe on the router or a domain controller? You could try adding a custom DNS record for dewarstaging.com pointing to the LAN address of the server. That should make it work internally, while leaving any external DNS untouched.

Or do it with hosts files on all the internal PCs, if there aren't too many of them.

Also, are you using modem and router interchangeably here? Just checking, because they're different things, which I'm sure you know.

While I don't know much about American ISPs, I've heard they do some wierd shit. They're the reason things like frame-relay and ISDN were still included in Cisco Systems exams until fairly recently. Blocking port 22 inbound seems unecessary. It's not as if it's an uncommon port.

Edit: Basically, I recommend a Fortigate. Or a Palo Alto. Or a Checkpoint. Or almost anything business-oriented. Though probably not a Cisco.
EDITED: 3 Feb 10:10 by PATCH
From: CHYRON (DSMITHHFX) 3 Feb 15:15
To: patch 20 of 24
Wish we had the SmartRG modem-router used by the previous ISP! It handled this hairpin stuff so transparently, I never even heard of it before, and didn't think to ask for it.

Coupla stories about that ISP: some time ago we needed to re-nat WAN port 80 to a different LAN IP, and I put in a ticket for the change. It transpired that in the course of changing ownership from local to multinational, they *lost* our router webmin login, and the one they had didn't work (note they had outsourced their support to India). Is that bad? Whelp, I was able to *guess* the login, and go in and update the config!

Now for the latest: a 'data hotel' subcontractor had a fire and explosion wiping out their optical fiber backbone (big-ass wire, whatever). In the middle of the worst blizzard we've seen in 30 years. It took them north of two days to restore service (by rerouting), and another two days to repair the cable. Force majeure, but my boss didn't like it one bit. So here we are ...

Suffice to say, configuring a 3d-party router for incoming ADSL (or whatever) is above my paygrade, and probably wild overkill for a *small* business?

Your idea about an internal, custom DNS record is plausible, maybe I can get my head around that. I *think* the aforementioned managed switch supports DNS somethings.

You the big dog!  :-D