Netgate Store

Author Topic: ATT Uverse RG Bypass (0.2 BTC)  (Read 5451 times)

0 Members and 1 Guest are viewing this topic.

Offline random003

  • Newbie
  • *
  • Posts: 10
  • Karma: +1/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #15 on: January 13, 2018, 12:20:24 am »
Do you have a bitcoin address? I'd like to give you $100 worth right now to keep the progress going. I don't mind a kernel patch.

Offline Ryu945

  • Full Member
  • ***
  • Posts: 139
  • Karma: +2/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #16 on: January 14, 2018, 10:34:44 pm »
I see that ATT uses IP-DSL to do the log in.  I wonder if there is a way to get Pfsense to do the log in directly.

Offline rajl

  • Jr. Member
  • **
  • Posts: 41
  • Karma: +0/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #17 on: January 18, 2018, 12:00:06 pm »
Do you have a bitcoin address? I'd like to give you $100 worth right now to keep the progress going. I don't mind a kernel patch.

I'm doing this for the love and the technical challenge, but tips are always welcome!  ;)

My bitcoin wallet address for any donations towards this project (from you or any other generous souls) is:

1H8CaLNXembfzYGDNq1NykWU3gaKAjm8K5

If anyone is going to tip me for my efforts, I figure I should at least give them something in return. 

Here’s the Netgraph based solution I have so far (which I have tested more thoroughly than the kernel patch):

Step 1: Copy the ng_etf.ko module from a FreeBSD 11.1 system to /boot/kernel.  If you are security conscious, you will copy this module yourself from a FreeBSD 11.1 system you already own/control.  For convenience, I have uploaded a copy of the module here which trusting souls may download at their convenience. 

Step 2: Add the following line to your /boot/loader.conf file – ng_etf_load=”YES” (include quotation marks)
Step 3: Reboot so that the kernel modules are loaded
Step 4: Clone the MAC address of the RG to your “WAN” port.
Step 5: Use the following Netgraph commands to create the EAP Netgraph Bridge.  As of right now you need to enter these commands at the console or include them in a startup/boot script.  Because you are disconnecting and reconnecting your Ethernet interfaces to create the necessary graph, you will lock yourself out of your own box if you are doing this over SSH.  Hence, you need physical access to your machine when entering these commands or they need to be executed automatically on boot.
Code: [Select]
## Replace “em0” and “em1” with your WAN and ONT Ethernet interfaces
    ## as appropriate for your machine

    ngctl mkpeer em0: etf lower downstream
    ngctl name em0:lower waneapfilter
    ngctl connect waneapfilter: em0: nomatch upper

    ngctl mkpeer em1: etf lower downstream
    ngctl name em1:lower laneapfilter
    ngctl connect laneapfilter: em1: nomatch upper

    ngctl connect waneapfilter: laneapfilter: eapout eapout

    ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
    ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'

If these commands throw a cryptic error message, one of three things happened.
  • Most likely, you omitted or added a colon (“:”) somewhere it didn’t belong (I speak from experience). 
  • Less likely, but possible, is that I accidentally omitted or added a colon while copying and pasting these commands into this writeup.  :)
  • Possibly, PFSense’s ng_eth module is not recognizing one or more of your interfaces.  This happens on some, but not all computers.  I’m not sure if this is a software or hardware bug.  For example, on my ZOTAC CI323, all of my interfaces are recognized.  However, on my QOTOM machine, only 2 of the 4 are recognized (OPNSense and vanilla FreeBSD recognize all four, which leads me to believe there is a subtle PFSense bug rather than an hardware issue).
 

Step 6: Connect the ONT to your PFSense Box and the RG to your PFSense Box (connecting from PFSense to the ONT port on the RG)

Step 7: Power cycle the RG in order to force authentication with ATT

Step 8: Confirm authentication.  After 1-2 minutes, you will see the “Broadband” light on your RG flash green and then go to solid green for a short period of time.  This means that the 802.1X port authentication has completed successfully.  However, your Broadband light will then start flashing read and then go blank.  This is because the RG is not receiving an IP address from the ATT network via DHCP (your PFSense Box is attempting request and receive the IP address).

At this point, I can see the DCHP requests and responses between the PFSense box and the ATT network using tcpdump.  However, the PFSense box is currently unable to use the IP address provided by ATT.  I assume this is because ATT is tagging all responses as being on VLAN 0 (you can see this in TCPDump).  With Linux based solutions, you can solve the problem by assigning vlan 0 to your WAN and then moving all your services over to the virtual interface created for vlan0.  However, FreeBSD doesn't handle frames explicitly tagged as vlan 0 very well. 

I submitted a bug report to FreeBSD for to see if I could get this resolved.  The short answer is "no" but they suggested that the Netgraph VLAN code might be able to convert frames tagged as vlan 0 into untagged frames prior to forwarding them up the network stack.  I have not had time to investigate this, but it seems promising.  If it works, I'll update my netgraph script above to incorporate the appropriate ng_vlan nodes.

EDIT - UPDATED Link for Kernel Module
« Last Edit: April 30, 2018, 12:56:07 pm by rajl »

Offline rajl

  • Jr. Member
  • **
  • Posts: 41
  • Karma: +0/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #18 on: January 18, 2018, 12:09:43 pm »
I see that ATT uses IP-DSL to do the log in.  I wonder if there is a way to get Pfsense to do the log in directly.

Unfortunately, no.  ATT uses cryptographic certificates installed in the ROM of the RG to authenticate the RG with the ATT network.  This allows them to prevent "unauthorized" equipment from being attached to the network.  Unless you feel like dumping the contents of the ROM, identifying the cryptographic certificates, uploading them to your PFSense box, and creating a custom authorization script, it's easier to just perform a man-in-the-middle attack on the 802.1X authentication.

Offline random003

  • Newbie
  • *
  • Posts: 10
  • Karma: +1/-0
    • View Profile

Offline rajl

  • Jr. Member
  • **
  • Posts: 41
  • Karma: +0/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #20 on: January 24, 2018, 08:43:16 am »
Thanks for your work on this.

https://tradeblock.com/bitcoin/tx/5be26573726e21c9f70d18af1223fb8e307cae6194a656ca294cc4afa99ae767

Received.  Thanks!

Digging into this a little more, the VLAN netgraph node may provide the "missing link."  I'm hoping to be able to test it in the reasonably near future and see if it actually works.

Offline aus

  • Newbie
  • *
  • Posts: 7
  • Karma: +1/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #21 on: February 14, 2018, 12:52:05 pm »
Hi rajl.

For the past few weeks, I've independently gone down similar avenues to explore a solution to this problem. I considered porting the EAP Proxy script, but that seemed painful. I opted to patch the kernal (if_bridge.c to be specific) to remove the drop on the 802.1X MAC 01:80:c2:00:00:03. That worked, but only for the EAP problem.

After discovering this post (a great start btw!) and reading more about netgraph, I agree that is probably the best approach. I hoping we can get this working using a combination of: ng_ether, ng_etf, ng_vlan, ng_tee and ng_eiface

For the VLAN0 problem, I expect the netgraph to look like something like this:

Code: [Select]
# em0 - ATT RG
# em1 - ONT
# em2 - LAN
# ngeth0 - "WAN" netgraph creates interface, removes VLAN0 tag from ONT traffic

# make eth devices addressable in netgraph
# (kernel module may already be loaded for you)
kldload ng_ether     

# from em1, create a vlan peer
# connect em1's lower hook to vlan's downstream hook
ngctl mkpeer em1: vlan lower downstream

# name peer vlan
ngctl name em1:lower vlan

# connect em1's upper hook to vlan's nomatch hook
ngctl connect em1: vlan: upper nomatch

# from vlan, create eiface peer (ngeth0)
# connect vlan's untagged hook to eiface's ether hook
ngctl mkpeer vlan: eiface untagged ether

# instruct vlan: to send vlan0 traffic to untagged hook
# which gets sent to the eiface ether hook (ngeth0)
ngctl msg vlan: addfilter '{ vlan=0 hook="untagged" }'

I've tested in locally in a VM and I think this part is working. However, the problem I'm struggling with now is combining the EAP netgraph solution with the VLAN netgraph solution. I think this is where ng_tee comes into play, but I'm still trying to wrap my head around it.

I think we need to use ng_tee to split out the ng_ether-em1 interface. Then hook up the EAP graph to one side and the VLAN graph to the other. But my head spins trying to keep left, right, right2left and left2right straight. :) The lacking documentation about netgraph doesnt help either. It seems no one talks about netgraph much.

Have you had any progress or success?

Offline aus

  • Newbie
  • *
  • Posts: 7
  • Karma: +1/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #22 on: February 17, 2018, 11:47:10 pm »
I just wanted to report back that I got this working, but probably not in the sense you were hoping for.

I’m running pfSense in a VM on Proxmox (KVM/QEMU). For now, I’ve opted to let the hypervisor (Linux) do the EAP and VLAN work. (Same method basically) Here is my setup:

Nothing too special configuration was required for my pfSense VM. Here is my config:

Code: [Select]
balloon: 0
bootdisk: ide0
cores: 2
cpu: host
ide0: ssd0:vm-100-disk-1,size=32G
memory: 512
name: pfSense
net0: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0
net1: virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr1
numa: 0
ostype: other
serial0: socket
sockets: 1
tablet: 0

The net0 (LAN) interface bridges to vmbr0. vmbr0 bridges to physical eth0, which is connected to my switch.

The net1 (WAN) interface bridges to vmbr1. vmbr1 bridges to vlan0. The vlan0 interface is configured off physical eth1, which is connected to the ONT. net1 MAC address also matches my ATT Gateway. Change it in your pfSense WAN interface setting.


/etc/network/interfaces:

Code: [Select]
# LAN / eth0                                                                                                                                                                                         
# Connect to switch
iface eth0 inet manual

# ONT / eth1
# Connect to ONT box outside
iface eth1 inet manual

# RG / eth2
# Connect to ATT Gateway on ONT port
iface eth2 inet manual

# LAN Bridge / br0
# Bridge main switch to pfSense
# IP is Proxmox host
auto vmbr0
iface vmbr0 inet static
        address  192.168.1.2
        netmask  255.255.255.0
        gateway  192.168.1.1
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0

# VLAN0 Bridge / br1
# Bridge vlan tagged WAN to pfSense
auto vmbr1
iface vmbr1 inet manual
        bridge_ports vlan0
        bridge_stp off
        bridge_fd 0

# EAP Bridge / br2
# Bridge ATT Gateway + ONT so EAP/802.1X auth can complete
# group_fwd_mask makes sure 802.1X traffic is bridged
auto vmbr2
iface vmbr2 inet manual
        bridge_ports eth1 eth2
        bridge_stp off
        bridge_fd 0
        post-up echo 8 > /sys/class/net/vmbr2/bridge/group_fwd_mask

Unfortunately, Proxmox conflicts with the vlan debian package, so you have to configure the vlan interface with the ip command instead of the interface file:

Code: [Select]
ip link add link eth1 name vlan0 type vlan id 0

And that’s pretty much it. I haven’t nailed down the timings yet from cold boot to online for a fully automated solution. For some reason, the EAP only takes under certain conditions. I have the best luck with the following:

1. Cold boot hypervisor
2. Wait for EAP to authenticate
3. Start vlan0
4. Start pfSense VM

It’s not perfect right now and it will take some more experimenting. But, it feels good to be off their RG!

 I’d still be interested in a pure BSD solution though.
« Last Edit: February 18, 2018, 02:28:44 pm by aus »

Offline rajl

  • Jr. Member
  • **
  • Posts: 41
  • Karma: +0/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #23 on: March 19, 2018, 09:24:12 pm »
Glad to see someone else exploring this.

I haven't had a chance to test it yet (my wife and toddler will kill me if I cut the internet tinkering on the weekend -- toddler cartoons are sacred). 

I hadn't thought of using an ng_tee node, but it sounds like a good idea.  Basically, you would connect the ng_tee left hook to the ng_eth downstream hook.  You would connect the ng_tee right hook to an ng_vlan (for example), which can filter the vlan 0 tagged traffic for you before passing it up the protocol stack.  The ng_etf could be connect to the left2right hook of the ng_tee node using the commands I wrote above to filter the eap traffic.

A modified script would look something like this:

Code: [Select]

    ngctl mkpeer em0: tee lower left
    ngctl name em0:lower T
   
    #Connect vlan to virtual interface for vlan0 traffic, ignore all untagged traffic
    #by failing to connect the vlan to the ether upper hook (reserved for eap filtering)

    ngctl mkpeer T: vlan right downstream
    ngctl name T:right vlan
    ngctl mkpeer vlan: eiface vlan0 ether
    ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }'

    #Connect other hook of T node to ng_etf node for eap filtering/proxying
    #Leave "lan filter" the same because we only care about eap traffic

    ngctl mkpeer T: etf left2right downstream
    ngctl name T:left2right waneapfilter
    ngctl connect waneapfilter: em0: nomatch upper

    ngctl mkpeer em1: etf lower downstream
    ngctl name em1:lower laneapfilter
    ngctl connect laneapfilter: em1: nomatch upper

    ngctl connect waneapfilter: laneapfilter: eapout eapout

    ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
    ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'


This is an untested script and may throw errors or error message, but the underlying principles should work for any brave souls willing to try! ;)  Just make sure that you have the ng_etf, ng_vlan, ng_eth, ng_eiface and ng_tee modules loaded.

Offline rajl

  • Jr. Member
  • **
  • Posts: 41
  • Karma: +0/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #24 on: March 26, 2018, 08:43:42 am »
Just occurred to me as an alternative that you could use a NetGraph multiplexer node instead of the ng_tee node.

Offline aus

  • Newbie
  • *
  • Posts: 7
  • Karma: +1/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #25 on: March 27, 2018, 12:12:47 am »
I did a bit more testing, but no success just yet.  (I suspect I need to first try to get this to work on phsyical hardware. Currently trying using a pfSense VM and I'm not seeing packet carry over from Linux hypervisor to the pfSense VM)

Regarding the ng_ether "bug", I did some digging on this. It turns out that this is not a bug necessarily. pfSense actually does a NGM_ETHER_DETACH against interfaces under some circumstanaces.

https://github.com/pfsense/pfsense/blob/9a18ac7af8ae4a4fde8998c18cc7ba7802056477/src/etc/inc/interfaces.inc#L180

I think this was for performance reasons back when netgraph had performance overhead.

Anyways, you think you'd be able to just do a control message of  NGM_ETHER_ATTACH, but that doesn't exist in vanilla FreeBSD. Luckily, pfSense integrates some patches to enable  NGM_ETHER_ATTACH, but you have to call it from PHP.

https://github.com/pfsense/FreeBSD-ports/blob/e178a5cf520e928efb3c7d896e3d9fcfb41ac7e5/devel/php56-pfSense-module/files/pfSense.c#L3094

This will re-enable the interface as a node in netgraph:

php -r 'pfSense_ngctl_attach(".", "em0");'

Also, for ng_one2many (assuming that's what you mean by multiplexer) I don't think that will work. I initially looked at this too, but it distributes packets in a round-robin fashion so the many's would only see some packets.  At least, that's how I interpret the man page.

Offline rajl

  • Jr. Member
  • **
  • Posts: 41
  • Karma: +0/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #26 on: March 28, 2018, 10:36:18 am »
ng_one2many is what I was referring to.  The man page states that it has several transmission modes, including round-robin and transmit-all.  The man page says that the round robin mode is the default, but my experience when playing with it is that transmit-all was the default.  In either case, you could easily set the transmission mode to transmit-all to ensure that you get the desired behavior.  So it will work and is simpler to work with than ng_tee.

That's great research on the ng_ether issue.  It's been holding me up for awhile, forcing me to do my testing on other distributions (e.g., vanilla FreeBSD and OPNSense) and then curse when I couldn't get it working on PFSense.  I'll have to see if it works with my scripts on a VM or PFSense.  It should, but Murphy's law always strikes me down!




Offline aus

  • Newbie
  • *
  • Posts: 7
  • Karma: +1/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #27 on: March 28, 2018, 03:53:37 pm »
Doh! Good catch on the ng_one2many transmit-all algorithm. I was looking at an old man page from an earlier version of FreeBSD, which it didnt support transmit-all yet. That's what I get for googling the man pages, instead of reading them in terminal! May give this a shot later... I'll report back if I have any success.

Cheers!

 

Offline aus

  • Newbie
  • *
  • Posts: 7
  • Karma: +1/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #28 on: March 29, 2018, 08:20:39 pm »
It worked!! True U-verse bridge mode on pfSense!

Code: [Select]
[2.4.2-RELEASE][root@pfsense.knox.lan]/root: ngctl list
There are 9 total nodes:
  Name: T               Type: tee             ID: 00000021   Num hooks: 3
  Name: ue0             Type: ether           ID: 00000003   Num hooks: 2
  Name: vlan0           Type: vlan            ID: 00000024   Num hooks: 2
  Name: <unnamed>       Type: socket          ID: 00000006   Num hooks: 0
  Name: ngctl96372      Type: socket          ID: 00000047   Num hooks: 0
  Name: ngeth0          Type: eiface          ID: 00000027   Num hooks: 1
  Name: waneapfilter    Type: etf             ID: 0000002a   Num hooks: 3
  Name: laneapfilter    Type: etf             ID: 00000031   Num hooks: 3
  Name: em0             Type: ether           ID: 00000019   Num hooks: 2
[2.4.2-RELEASE][root@pfsense.knox.lan]/root: ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=40098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWTSO>
ether xx:xx:xx:xx:xx:xx
hwaddr xx:xx:xx:xx:xx:xx
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
[2.4.2-RELEASE][root@pfsense.knox.lan]/root: ifconfig ue0
ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
ether xx:xx:xx:xx:xx:xx
hwaddr xx:xx:xx:xx:xx:xx
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
[2.4.2-RELEASE][root@pfsense.knox.lan]/root: ifconfig ngeth0
ngeth0: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=28<VLAN_MTU,JUMBO_MTU>
ether xx:xx:xx:xx:xx:xx
inet xx.xx.xx.xx netmask 0xfffffc00 broadcast xx.xx.xx.xx
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

For reference...

em0 is connected to my ONT.
em1 is connected to my LAN
ue0 is connected to my RG (via USB ethernet)
ngeth0 is the VLANed device which is configured as my WAN in pfSense


Commands to get it running (thanks for the help on ng_tee rajl!)  ...

Code: [Select]
# copy and load ng_etf kernel module

kldload /boot/kernel/ng_etf.ko

#
# setup netgraph nodes
#

# list out netgraph nodes

ngctl list

# pfSense for some reason detaches ether devices. reattach any missing devices.

php -r 'pfSense_ngctl_attach(".", "em0");'

# create tee node to split em0 traffic (one for EAP, one for VLAN0)
ngctl mkpeer em0: tee lower left # may get a warning
ngctl name em0:lower T

# create vlan node + eiface
ngctl mkpeer T: vlan right downstream
ngctl name T:right vlan0
ngctl mkpeer vlan0: eiface vlan0 ether
ngctl msg vlan0: 'addfilter { vlan=0 hook="vlan0" }'

# create etf and connect to em0 (ONT)
ngctl mkpeer T: etf left2right downstream
ngctl name T:left2right waneapfilter
ngctl connect waneapfilter: em0: nomatch upper

# create etf and connect to em1 (RG)
ngctl mkpeer ue0: etf lower downstream
ngctl name ue0:lower laneapfilter
ngctl connect laneapfilter: ue0: nomatch upper

# define filters for EAP traffic
ngctl msg waneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'
ngctl msg laneapfilter: 'setfilter { matchhook="eapout" ethertype=0x888e }'

# use filters to bridge EAP traffic
ngctl connect waneapfilter: laneapfilter: eapout eapout

# change MAC address to match RG (also can be done in pfSense)
ifconfig ngeth0 ether xx:xx:xx:xx:xx:xx


There is still worked to be done though to make this perfect though...

1. Explore using ng_one2many to see if that simplifies the netgraph a bit
2. Automate /  Harden change so its persistant across reboots (rajl already documented this earlier)
3. Document!

And for what it's worth, I'm running this pfSense in a virtual machine via Proxmox (QEMU/KVM). I couldnt get the VLAN0 traffic to bridge across the interface into pfSense, so I ended up doing a PCI passthrough of the NIC device.


Offline rajl

  • Jr. Member
  • **
  • Posts: 41
  • Karma: +0/-0
    • View Profile
Re: ATT Uverse RG Bypass (0.2 BTC)
« Reply #29 on: March 29, 2018, 08:44:24 pm »
That's awesome!!!  My suspicion is that this would run on baremetal just fine (have to test though).  So let's say there's a 4th todo - test this to run on baremetal for those of use that don't virtualize! :-)  Hopefully, it won't take too much modification.


This should be pretty easy to automate so that it executes across reboots.  Just save your commands as a shell script (don't forget the #!/bin/sh at the beginning of the file) and follow the PFSense instructions for executing shell scripts at the end of the boot process.

https://doc.pfsense.org/index.php/Executing_commands_at_boot_time

I read somewhere that ATT will occassionally push firmware updates to the RG, which this setup may have problems with because the RG is being isolated from the ATT network.  But that's a bridge to cross when we get there.