Tech Geekery Inside (Linux server, unable to connect via remote)
 

So, I've had various Linux servers running on this network for years. Since several people on here claim to be big Linux geeks, let's see if any of you can come up with things I've missed.

Recently, I've had a previous server running, but between a recent ISP change (A local ISP bought my old ISP out...), plus due to a few other factors (Moving...), my personal server's been down during this ISP change.

Everything is the gorramn same on my network. I've gone so far as to DMZ my Linux server. It's running a stock Ubuntu 12.04 Server install with webmin, LAMP, Java/gcc/g++/make/etc., and Samba.

I can ssh, connect via http/ftp/sftp/whatever perfectly fine if I'm in the local network. But as soon as I do any kind of remote connection, even from computers in the local network to my DynDNS host redirect or even directly connecting to my WAN IP address (Remember, it's DMZ'd, so it should be fine).... I get "connection timed out", as if the thing was behind my router and wasn't DMZ'd/et al'd.

At this point, I'm stumped. iptables even returns a standard setup. I want to start pointing fingers at this new ISP blocking any remote connections or something, because port forwarding doesn't work, virtual servers don't work, DMZ does not work, and it's a stock gorramn Ubuntu install that I just finished installing a little bit ago. Here's the kicker - ping works fine. The server responds to any ping from anything anywhere fine.

Thoughts? I'm freaking stumped after struggling with this most of the day. My hunch is that it is related to some obscure change in 12.04 that I missed/fatfingered or it Really Is just my retarded ISP.

Too few details. DSL/Cable? Which router? Do you have a static or dynamic IP? What private subnet are you using? How many eth interfaces on the linux server? A topology diagram would help.

What happens if you turn off iptables on the ubuntu server and try to connect?

You mean flush the tables and set the default policy to ACCEPT?

I mean that would work to. I am just semi lazy and "#service iptables stop" test connection then "#service iptables start" would work. By no means is that the most secure way to do it, but it should work. I also have not spent to much time using Ubuntu as a server. I am mostly a RHEL guy so there might be a few differences.

Neither, fiber. Dlink DIR-655V2. Dynamic, hence dynDNS. 192.168.0.* (LAN only network) and 192.168.1.* (WAN-connected network, the Linux server is only hooked up to this). One eth interface.

Topology is basically "Connect Linux server to router. Connect Router to internet".

It is set that way already.

Already tried that. First thing I tried in fact, SHOES!

Thanks guys. Still stumped, I'm baffled and confused. I've been setting up Linux servers for a longer time then most would believe, and Ubuntu from 9.x to 11.10 worked perfect for me. This new server running 12.04 is blowing chunks for me so far though. What baffles me is that the connection is /timing out/ instead of being refused. If I un-DMZ/un-port forward/etc. everything, it just results in connection refused at the router itself.

Port forwarding seems like your problem. Are you using a new router or now two routers?

I have an Ubuntu 11.10 box acting as a router for my internal (LAN-only) network.

The external (WAN) network is the D-link, the new server shouldn't be affected by the LAN-only network in any way, shape, or form. It is not connected or otherwise touching my internal network. Notable note: Once I get this server working properly, it's replacing the D-link as a router for my WAN network.

I've gone so far as to disable port forwarding and just enable the DMZ, Sam. The second it was DMZ'd, that should have ended any router-specific problems.

Traceroute from outside and see where the hops stop.

1 75.125.232.57 (75.125.232.57) 0.509 ms 0.642 ms 0.571 ms
2 te1-4.dsr01.hstntx1.networklayer.com (207.218.223.5) 0.583 ms 0.448 ms 0.367 ms
3 po16.dsr02.hstntx2.networklayer.com (70.87.253.105) 0.907 ms 0.783 ms 0.824 ms
4 ae17.bbr02.sr02.hou02.networklayer.com (173.192.18.238) 0.543 ms 0.691 ms 0.532 ms
5 ae3.bbr02.eq01.dal03.networklayer.com (173.192.18.220) 10.407 ms 10.294 ms
6 ae7.bbr02.eq01.dal03.networklayer.com (173.192.18.209) 8.017 ms 8.006 ms 7.887 ms
7 ae1.bbr01.cs01.den01.networklayer.com (173.192.18.139) 22.590 ms 22.541 ms 22.401 ms
8 six.(secret).com (206.81.80.173) 61.656 ms 60.327 ms
9 206.130.137.1.(secret).com (206.130.137.1) 55.700 ms 58.571 ms 57.624 ms
10 CC-3-DHCP-96.46.18.40.(secret).net (96.46.18.40) 61.643 ms 59.321 ms
11 CC-3-DHCP-96.46.18.40.(secret).net (96.46.18.40) 67.827 ms 63.317 ms 58.880 ms
12 CC-3-DHCP-96.46.18.40.(secret).net (96.46.18.40) 59.146 ms * 59.431 ms
13 CC-3-DHCP-96.46.18.40.(secret).net (96.46.18.40) 59.128 ms 67.484 ms 59.139 ms

Nothing drops, it responds correctly to both ping and a traceroute Rev.

Does your router have an http interface, and can you connect to it from outside?

Yes, Yes if I enable remote management.

That means that this particular port is left accessible by your upstream ISP. Shut down the remote management, and setup apache on your server on that particular port. Then setup portforwarding in your router, so that particular port is forwarded to your linux box. Test the apache from inside first, then outside.

What does your hosts.allow and hosts.deny files say? Also what does "#ssh -vvv server" tell you when you try to connect?

:80, :8080, :8181 are all blocked somewhere outside of my router if I enable remote management and direct it to those ports.

(Edit) I take it back. Inteeeresting, :8181 is not blocked now. Setting up a virtual server on port 8181 to forward to 80 on the new server.

Remote management enabled, set to port 80 or 8080: Timed out.

Remote management enabled, port 8181: Connects fine.

Port 8181, set as a virtual server to forward to port 80 on the new server: "Problem Loading Page, unable to connect". No longer timing out.

The hosts.allow and hosts.deny are all blank.

ssh -vvv server from a server remote to my network (SSH to remote server, to SSH back to the network) gives a (edit, correction) connection refused error.

Timing out is an indication that packages are actively being droped somewhere. Where as a connection denied means that the host is replying that a service ia not setup on the server (unless of course the firewall is configures to REJECT rather than DROP).

Can you connect to 80 from localnet?

The server works perfectly if I connect from my 192.168.1.* IPs.

The SQL server, apache, even my gorramn custom-written servers all connect perfectly. If, however, I use 96.46.21.149 to connect instead of 192.168.1.2 it all goes to shit.

Please remember that everything was set up identically two weeks ago and worked perfectly except for A) a new server box running 12.04 instead of 11.10 Ubuntu, and B) An ISP change due to being bought out. My router isn't either option. It hasn't changed since it worked perfectly with my setup 2 weeks ago.

I dont know about your router, but on mine if I hit my public IP from the inside, portforwarding works fine. Can you test portforwarding with a production server?

No. This is my home network.

I'm going to try to flash to newer firmware and restart everything from scratch in the router over the next ~15 minutes. If nothing changes, I'm assuming it's not router.

(Edit) firmware *is* the latest version, resetting router to factory defaults.

Set up router to bare basics. Set a static IP to the server (192.168.1.2)

Connection times out on everything I try to do to it. Including ssh -vvv to 96.46.21.187.

Remote Management works fine on port 8181. Nothing else does.

Call the ISP? Or are you technically not allowed to port forward? I would think if you are paying for a fiber line you can do pretty much whatever the fuck you want with it. Who runs the line? Verizon Fios?

Last resort here. The new ISP *sucks*, and I was with my old ISP specifically to not deal with them.

Yeah, these guys think residential lines are *only* for web browsing. They baaaaad.

Nope. Local telecom company that leases fiber from our local utility district.

If it wasn't for the retarded shit related to servers with these guys, they could put Verizon FIoS to shame. But they go FULL FUCKING RETARD in many aspects since the owner doesn't really know wtf he is doing - this is why I wasn't dealing with them before.

I'm going to keep fiddling, but I'm having difficulty believing it's my router when, two weeks ago, this router worked perfectly. I'm also having difficulty believing that it is the server, 'cause I've dealt with Linux servers for over 15 years ('94? '95? '96? Somewhere in there.) so...

If it works inside, doesn't work outside, and the ISP is a bitch who thinks they can control what you run, it is probably their fault.

That's the bad news.

The good news is, if it is a local telecom that rents the fiber, there is probably an easy way around it. The trick will be finding it.

Figure out which ports some of those fancy security systems are running on, and tell the local ISP you need them open. Then run everything through those. Or hell even make up some ports and tell them your security system needs them open. Or some other legitimate need for open inbound ports, as that was the first thing that popped in my head.

I'm just trying to get this setup for when I finish my move across the state in a week. I won't have any relation to them very shortly.

Drives me nuts that I can't thoroughly test any of this /before/ I finish my move. I /hate/ being unprepared like that.

If pf isnt working from the inside, its not the ISP's fault.

Everything works fine from 192.168.* addresses, Rev. Trust me on that, I've tested it extensively.

Bleen.

What happens if you put another box in the DMZ and configure it for connection happiness.

Valid question Faefae. Trying that now.

P.S. If Faefae outsmarts us all.....

Now THIS is interesting.

When I try to visit port 80 on the WAN IP to the local webserver on this machine, all port 80 traffic is temporarily blocked. Including my traffic to/from MT.net - yes, I'm getting MT.net timeouts when I try to access the server on this machine via the WAN IP, and not the LAN IP. If there's no port 80 traffic bound to this machine's webserver, I get perfect MT.net access. If there is port 80 traffic bound, I even have sites like google and microsoft timing out - but if there is none, everything works perfectly.

Hmmmmmm.

(Edit) And interestinger it gets. Inbound port 80 traffic blocks port 80 traffic completely to all my machines - send or receive.

Yep, it's the ISP.

Inbound traffic of a certain type (Email, SSH, web server, etc.) gets a block put on that port and no data is sent or received untill the block times out. Doesn't even matter if it originates from my internal network or if it's external, it still blocks it. This behavior has also been verified on a friend's network that subscribes to the same ISP. So, unless both of our routers (His is a real, genuine Cisco, mine is a d-link) have the exact same bug, it's the ISP doing it.

Bad ISP is bad.

I don't get it. How does your ISP know about internal traffic (192.168 -> router -> 192.168) when the traffic never leaves the router? Hitting a PFed port on the public IP of the router from the inside, your packages should go no further than the router itself, and then back to the internal network. Something is wrong here.

I don't know.

The same setup worked fine at my new apartment the last time I was there, Rev - same computer even.

I wish I had an answer for you, but simply by changing the ISP, it works now.

And ran into one final headache. I have everything working perfectly now, except for one hiccup. I cannot get a wireless AP to handle traffic properly. If I connect it to the network, the clients can get the DHCP address properly, and can run everything perfectly in-network, but external network it refuses to forward traffic from the Linux router.

It's as if I did not use the iptable command iptables -t nat -o eth0 -A POSTROUTING -j MASQUERADE in fact. Everything works perfectly except for passing traffic through eth0. eth1 and eth2 interact perfectly. I'm stumped, my switch works perfectly. But even if I set my wireless AP to full AP operation (Basically, wireless switch mode), it still won't pass wireless traffic through to eth0 - regardless of if it is connected to my switch on eth1 or eth2.

Any ideas guys? I'm having trouble figuring this out.

(Edit) Now this is very, very odd. I'm seeing a 500ms or so ping to even Google. Something's not right here.
(Edit2) Fuck the previous edit. Steam decided to download a shit ton of stuff without telling me. Non-relevance is not relevant.

Can you ping your router's external IP from a machine connected to the AP? Can you ping the router's remote peer? Does the AP has its own DHCP server? If so, does it assign IPs on the same subnet and does it assign a proper gateway?

There is no router now, Reverant. Just a Linux PC acting as a router which I've connected gigabit switches to.

No - No - Yes and no, I can set it to either and have - Yes if I set it to assign IPs to the correct subnet, but I can set it for pure dumb switch operation too.

(Edit) Samba, SSH, all work perfectly to the internal network if using either ethernet interface on a wired switch. I can't connect to the same from the external IP via wireless, although I am daisy-chaining switches in this case (wired switch->wireless AP/switch). I can do it fine from wired, however.

Well if the AP distributes IPs for a different subnet, and the AP is configured for dumb switch mode, its oviously not going to route to non-local IPs since there is no notion of a gateway in this setup?

Set it up to serve IPs on the same subnet as the wired net, and make sure the wireless clients get the proper gateway address.

If I set it to "smart" operation (assigning IPs), it is set to assign to the same subnet Rev. If I set it to dumb operation, it simply acts as a switch and passes IP assignment on to the Linux box.

Either method results in a big helping heap of fail in trying to get the wireless to pass traffic past the Linux box onto the internet.