The Holy Grail – Getting a Raspberry Pi Access-Point on the Lone WiFi Device… with Internet Passthrough and Android Working Too!

Leave a Comment / ESP8266, IoT, Raspberry Pi, WebServer / By / January 2, 2024

Edit January 2, 2024 – There were a few issues running this procedure on a Desktop version Raspberry Pi 4B, using the 64bit Bullseye. My main goal was for the Headless, Lite version on a Zero and it worked as I needed. Doing the same procedure on the 4B had issues with the two WiFi devices (the real versus the virtual). From the 4B Desktop, it would connect to my router, but I couldn’t access the Internet nor could devices connect through the Hotspot. It also didn’t passthrough the Internet to the devices. The changes below also marked in green, correct these problems.

Before I get too far along patting myself on the back I need to disclaim. I’m actually really new to Raspberry Pi’s and Linux in general. I merely spent the last three days beating Google to death. I stand on the shoulders of many experts and expert wanna bees. My research travels have also taught me that this area is in a constant state of flux. What people said would work on a Raspberry Pi 3 years ago doesn’t work for me now. What people say works, right now, on a Raspberry Pi 4, didn’t work for me now. What used to work on earlier Raspberry OS’s that used dhcpcd are totally broken under the newer NetworkManager that is available on Bullseye and is supposedly going to be the default on Bookwork. Things that are setup with the old mentality of dhcpcd are broken and mangled to death when converting to the new NetworkManager way of doing things. Even many experts seem to be on uncharted ground. And NO ONE seemed to be able to put all the pieces together. I’ve seen solutions offered with hundreds of lines of commands or running obscure scripts that are no longer supported… BECAUSE they no longer work. Many solutions wanted me to pay a fee to purchase some third-party software that promised to make my life easier, and better as would cook me breakfast. I bet of the hundreds of posts I’ve read and threads I’ve traversed over fifty percent said this absolutely required two network connections… either a WiFi and Ethernet, or… the built-in WiFi and another WiFi dongle. And NO thread, post, blog or website ended with a working solution for all the pieces working together! But because of all this guidance and miss-guidance, I want to be very clear that this solution is only known to work for the one use case that I care about and tested. I don’t know enough to know how portable it is and I didn’t test it on anything else. If it works on any other combination of equipment and/or software will be purely dumb, blind luck!

What It Will Do… When Done

  • Runs on a Raspberry Pi Zero W
  • Exposes an Access Point using the built-in WiFi
  • Connects to a router’s Access Point using the same built-in WiFi
  • Can expose a website to any device connected directly or coming through the Router’s LAN
  • If the connected LAN has access to the Internet, devices connected through the Zero’s Access Point can also access the Internet.
  • It will work with Android devices. Trust me, there are a lot of posts, threads and blogs that haven’t found a successful solution.

InqVault

My current project is to create a generalized LoRa Gateway that doubles as a database server and web server – InqVault. It will act as a conduit to the Cloud as well as expose browser based dashboards, charts, graphs, and tables to present real-time and historic data to the user for any manner of different fields. It is planned to have plug-in ability to leverage secondary analysis tools to do on-going topic analysis of data and present other streams of data. The sensor nodes will use the LoRa communications to allow for extreme long term and long distance data acquisition using batteries. For higher-rate, nodes with access to reliable power, MQTT data is expected to be supported. I’m really just writing it for myself, but I always, optimistically code it as if it’ll actually be useful for others. So again, the following procedure is based solely on this configuration. The base-line system will include:

  • Raspberry Pi Zero W – When dealing with transactions of tens or hundreds of bytes per hour, really… do I need a Raspberry Pi 5? Besides, the power utilization of this on a left-over USB power supply is less than a $1US of power per year.
  • Raspberry Pi Bullseye – Well… because Bookworm is full of bugs and doesn’t even run on a Pi Zero.
  • RFM95W LoRa Modules
  • ESP8266 Modules – Client nodes will use this ubiquitous micro-controller for its CPU power (80 or 160MHz, 32bit processor), 4 μA power utilization while in Deep Sleep and mainly because they can be had for less than $1US in relatively small quantities.

Speed Bump

In my hobby, I work with low-end microcontrollers, Arduino, ESP32, Raspberry Pi Pico and mainly ESP8266. It is brain-dead easy to get a $1 controller to host a website and expose it on a LAN via a router using a station connection WHILE ALSO exposing a Soft Access Point so someone can connect to it directly. When I did the 10,000 foot design for this project, I took that simplicity as a given. I expected a high-end Raspberry Pi and million-lines of code Linux OS to be able to do the same without causing a burp. Well… I come to find… that’s just stupid talk!

You see… if InqVault were a product line… I’d have these little Raspberry Pi Zero servers and I’d have a bunch of ESP8266 sensor nodes in the inventory. A customer would have to be able to configure these things. Now considering, I’m a Raspberry Pi Noob, I have to assume many of my customers may not even have heard of a Raspberry Pi or Linux. I can’t have them building these up. They have to be turn-key… or nearly so. So the VERY first thing they must do is to configure this server for their router. The only practical, user-friendly way I know to do that is to expose a Kiosk, webpage over an Access Point. The customer can simply connect their Android or iPhone, tablet, laptop, or desktop WiFi connection to the server’s exposed SSID and configure the server for their LAN. This trivially easy task on a a lowly ESP8266, is like pulling teeth on a Raspberry Pi and has turned my hair white for Christmas. So… without further gilding the lily…

What Pieces Are Needed

As mentioned above (I know you didn’t read it, I wouldn’t have) this hasn’t been exhaustively tested on all kinds of hardware and software combinations. Updates break things every day. Raspberry Pi 5’s break about everything. So these are the pieces and procedures I used that work. If you don’t follow things exactly, and get in a jam, I doubt, I’ll be able to diagnose your problem. Diverging because you know a trick on the console might just land you in La-La Land. I’ve read un-counted posts where some one-liner gets wiped-out or totally ignored because of NetworkManager. Now… I’d love to hear your one-liners, as I’m still learning every day. Here goes…

  • Raspberry Pi Zero W – Yes, the single core, first generation.
  • Raspberry Pi OS, Bullseye Lite – Because Bookworm isn’t available on a Zero. And… seriously, have you ever tried running the Desktop on a Zero?
  • Raspberry Pi Imager 1.8.4 – Hey… it’s the latest.
  • Microcenter 32GB Micro SD card – It’s just a development mule.
  • NetworkManager – Well… they tell me its the future!
  • No, gimmicks, no scripts, no extra software to install. It all comes in the Lite version.
  • Really… considering how many pages wanted me to try this or that… this is the simplest post you’ll find that does everything needed.

Without Further Ado…

I’m not going to give you a picture of every step with an arrow pointing to the button you need to press. I’m expecting this is not your first Rodeo and probably know more than me! If you need that much more detailed level of handholding, you’re looking in the wrong place. I will bullet things about as concisely as my cheat-sheet I wrote as I tried out things and subsequently when I re-did it without pause on a fresh SD. I’ll write the Linux commands you need to type in, in bold. It’ll probably have text after the bold part as explanation. I would want to know why I’m doing something. Don’t type these comments in! Of course, there will be places in the text where you’ll need to use your own data. I’ll mark those like in red, like – your SSID. Just change the red portion. If there are quotes around it in black… keep those.

Some of these steps, I found on the Internet. I’ll have references where I found them because they were really obscure. Many of the items, I saw in so many places, I forgot which one was where I triggered… “Hey! I need to do this!” Here goes…

SD Card Prep

  1. I usually create my Raspberry Pi’s completely headless during the install and definitely in usage. Unfortunately, this procedure has one step that breaks that. I’d love to hear a way around this, but it escapes me at the moment.
  2. Load up the latest Raspberry Pi Imager (I have v1.8.4 here in the last days of 2023)
  3. Choose your device – I have a Raspberry Pi Zero W
  4. Choose you operating system – I have chosen the Raspberry Pi OS (Legacy, 32-bit) Lite. (Debian Bullseye)
  5. Choose your storage (I have a 32GB Microcenter Micro SD card HC 10)
  6. Next
  7. Edit Settings
    • General – Fill out every section if you want to run headless
    • Services – Make sure Enable SSH is checked and use password authentication
    • Options – Knock yourself out, go wild!
  8. Press Yes, to apply your settings and write your OS to the SD card

Running on the Raspberry Pi

  1. Stick it in the Raspberry Pi (Dah!) and power it up (takes about 3 or 4 minutes)
  2. If you’re not watching it on a screen, access it with your favorite SSH: PuTTY, Windows Powershell or whatever you all use on Mac or Linux.
  3. Log in.
  4. sudo apt-get update – These are always obligatory.
  5. sudo apt-get upgrade – Make sure we have the latest and greatest (although down the road this will probably break things from this post).
  6. sudo raspi-config – There are many things you’ll probably want to do in here as I do, but I’ll only show those pertinent to this goal.
    • Advanced Options / Network Config / NetworkManager – This is where you’ll lose connectivity if you were using it headless. I found it rather rude while I waited for it to do something.

The Ah-ha Moment

At this point I had to hook up a keyboard and monitor… yeah, the prompt was there waiting for me to do something new.

About the third day of beating my head against the desk, I realized something. The ESP8266 can do what I want, so why is the Linux OS keeping me from doing the same thing… or worse making me use two NIC’s and bridge them. Something clicked… I could make an Access Point, but either the WiFi device (wlan0 in my case) could only be used as the Access Point… OR… it could be used to connect to the router. If I turned one on and the other would turn off. This was rather frustrating since I was back to working headless. Well… this is where most of the posts were telling me that I MUST add an Ethernet cable or add a WiFi Dongle. The OS was making sure that we only used the one device for one thing!

Can I make an alias/virtual device, point it to the same NIC and hope the OS doesn’t notice? After a bunch of new searches, I finally came across this lone topic that was doing what I wanted. https://blog.thewalr.us/2017/09/26/raspberry-pi-zero-w-simultaneous-ap-and-managed-mode-wifi/ Unfortunately, using the whole topic didn’t work for me. It was written long before NetworkManager came to Raspberry Pi and NetworkManager pretty much mangled everything to death… time to start over, build another SD card. But, the creation of the virtual device stuck with me. It appeared to me that although NetworkManager worked with devices, it didn’t seem to mangle them up like it does with connections and everything above. To add this virtual device, you’ll need to create a new file on your Raspberry Pi:

  1. sudo nano /etc/udev/rules.d/70-persistent-net.rules and save the following into it. Do it exactly. No new lines, no differences…
SUBSYSTEM=="ieee80211", ACTION=="add|change", ATTR{macaddress}=="b8:27:eb:ff:ff:ff", KERNEL=="phy0", \
  RUN+="/sbin/iw phy phy0 interface add ap0 type __ap", \
  RUN+="/bin/ip link set ap0 address b8:27:eb:ff:ff:ff"

… except note: Replace the red MAC address with that of your WiFi on your Raspberry Pi. Use ifconfig -a to find your MAC address. This threw me for a while… It was under the wlan0 section, but it was next to something labeled ether. This rules file was the first breakthrough for me. Save the file.

  1. sudo reboot – I had to reboot the machine for the OS to register it as a new device being equivalent (and yes, pointing to the WiFi’s MAC) but it was a separate device!
  2. sudo nmcli dev wifi hotspot ifname ap0 ssid <your AP SSID> password “<your AP password>“ – This creates a connection in NetworkManager… kind of like when you make a connection to your router’s SSID and password using the wlan0 device. In this case, you’ll note, it is using the ap0 device we created in step 7… which is simply an alias to the same wlan0 device. Magic happens!
  3. sudo nmtui – This is the console GUI for NetworkManager. At this point, we have made all the settings necessary for our AP, but it is not turned on. There is a command: sudo nmcli connection up Hotspot that will turn it on… BUT, when you reboot, it won’t automatically start up. Since this is a server, I want it to turn on at power up. We need to go into the GUI and configure the connection to automatically start.
    • In the GUI select – Edit a connection
    • Select – Hotspot. This is the connection you made in step 9.
      • In the IPv4 CONFIGURATION section, check Never use this network for default route
      • Move down to the bottom and check – Automatically connect
  4. At this point, you should be able to connect with:
    • Windows – However, if you try it now, Windows will ask for a PIN instead of the password. It does provide a link to enter the password instead. (more in a moment).
    • ESP8266 – Yeah!
    • But I get no love trying to connect to the Hotspot with Androids.

Making it so Androids can connect

  1. This can be fixed by making one small edit in the Access Point configuration file that you created back in step #9. Unfortunately, the GUI in nmtui doesn’t expose the value for changing it. Edit the file – sudo nano /etc/NetworkManager/system-connections/Hotspot.nmconnection. If you get a blank editor, you need to check in the directory. Maybe the file name is different based on settings you used. You should have two files – one for the connection to your router and the second should be the AP hotspot. Obviously… edit the hotspot.
    • In the file, down in the [wifi-security] section, there should be an entry proto=rsn; Simply change this to proto=wpa; and you should be golden with everything. Save the file. Important: On the Raspberry Pi 4B, if you later make changes with the Desktop NetworkManager GUI (found in the Taskbar) it will overwrite this setting and on the next boot, you’ll find things have changed.
  2. You’ll need a reboot… again.
  3. Now, if you try things out you’ll be able to connect to the hotspot with Android and Apple devices. You’ll also note that Windows will prompt you for the password instead of the pin now.

Conclusions

Most all posts from people, far more expert than I, were claiming that this absolutely required two NIC’s. Either the built-in WiFi and Ethernet or the built-in WiFi and a second WiFi dongle. I guess I didn’t know enough to know I should give up. The few posts I ran into that said it should be able to run on one, didn’t elaborated on how to do it. And fewer still said that even if you could, the performance would be horrible. In my case, I merely want this for a short period of time to allow an end-user to configure a server via a web interface exposed via the Access Point. The primary goal being to allow them to connect it to their router and LAN. However, it sounds like some people want to use the Raspberry Pi as a kind of router or a way to get around Hotel security or limits. Either way, my Inquisitiveness forced me to see how bad this might be. Unfortunately, where I live, my only Internet option for Internet service is Starlink. It is a little erratic when benchmarking. I have partial tree blockage on it and sometimes I miss satellites. I can do two tests in a row, and one be way over 100 Mbits/second and the second be under 50 Mbits/second. I’m not complaining… I think it is the best thing since Sliced Bread! My second place options never exceeded 1Mbit/second! Anyway… I did try comparing going directly to the Starlink router and via the Raspberry Pi Zero-W Access Point and then on to the same router. The speed runs were all over the place as I expected, but I think I did notice a significant hit.

  • Android Phone <-> StarLink Router = ~ 100 Mbits/second down / 19 Mbits/second up
  • Android Phone <-> Raspberry Pie Zero W <-> Starlink Router = 16.7 Mbits/second down / 17 Mbits/second up

For those of us living out in the wild, these are incredibly good numbers either way. You city-people used to multi Gigabit… well you have different expectations than us. In either case, these numbers are just fine for the purposes of merely configuring the connection to their router’s access point.

I hope this post has been useful for you. And by all means… if you know some ways of packaging this up to be easier to use, I would certainly be interested. I’m particularly interested in a way to do this totally headless. The losing of the connection when switching to NetworkManager back in step #6 is requiring me to break out a keyboard and monitor just for that one step. Everything before and after can be done headless. Good luck with your projects!

VBR,

Inq

Post Views: 82

Leave a Comment

Your email address will not be published. Required fields are marked *