Ubiquiti Edgerouter ER-X Flash issues (free space, hangs, firmware upgrades)

 

The ER-X is a very cheap 5 port router, with a lot of enterprise-level software features. It's almost too cheap, because it doesn't really have enough Flash memory, and certainly with earlier samples, the Flash itself was rather unreliable.

For a start, there isn't enough Flash space to store three images, so when you want to upgrade the firmware, you always need to delete any old backup version that's still installed to make enough free space ("delete system image" CLI command).

The firmware upgrade process ("add system image") itself can be rather hit&miss on devices that suffer from unreliable Flash, requiring multiple attempts to succeed. The symptom I found is an apparent hang at the "Copying Image" stage, with a crash and reboot after a few minutes.

There are many reports on the Internet of ER-X having random hangs with various software versions, in particular there's a symptom where IP forwarding seems to work normally, but you can't SSH into the router, and the DHCP server stops responding.  

My experience of this problem is that it's caused by the router's R/W Flash filesystem filling up.  The software uses Squashfs and Overlayfs to give a largely read-only filesystem, with just a few configuration files and additional software packages in the R/W area. 

The currently running filesystems are based in /root.dev. There will be a squashfs.img and a "w" directory that contains the R/W overlay filesystem. Any older firmware versions that are still installed will be in the same place but with a ".o" suffix.  This is quite handy because you can easily look at configuration files from an older version in the /root.dev/w.o tree.

Even if it looks like you have enough free space after doing an upgrade, there's a regular cron job that runs every 12 hours and updates the Apt cache. On my device, this job is sometimes enough to fill the filesystem up.  So I always need to remove the old installed firmware once I'm confident that the new firmware is working well enough.  Rather than properly deleting the old firmware, if you still want to retain an easy way of reverting, it's possible to just copy off the squashfs.img.o file; delete it, and then copy it back on  if you ever need to revert back to the earlier version.



Comments

Post a Comment

Popular posts from this blog

DECnet-VAX Phase V WAN connectivity with simh

Installing VMS and DECnet-Plus HP's VMS Hobbyist program used to make the VMS 7.3 distribution CD available as an ISO image file. While there is no longer an official hobbyist program, I would imagine that the CD image is findable if you look hard enough. Once in possession of an image of the VAXVMS073 CD, you can install it using your simh configuration of choice, which must have a Qbus or Unibus.  There are plenty of examples out there to help with VMS installation, but if starting from scratch, you boot from the CD; use standalone backup to expand saveset B onto your normal boot medium, then boot as normal and go through the installation dialogue.  During the installation you should choose to install DECnet-Plus (aka Phase V). Here's an example:  https://raymii.org/s/blog/OpenVMS_7.3_install_log_with_simh_vax_on_Ubuntu_16.04.html To make the system fully useful, you'll need a valid license loaded for at least VAX-VMS and DVNETEND (endnode) or DVNETRTG (router).  Once aga
Read more

Time Electronics 9814 Programmable Voltage Calibrator

Image
I left school having just turned 17 after my 'A' levels in 1980, needing to fill a year before going to University, and ended up working at Time Electronics in Tonbridge.   The company made electronic calibration lab instruments, and I worked there for a year before University, and then in most vacations subsequently until I started my first post-graduation job at DEC in 1984. More reminiscences about this in a future post perhaps. The biggest project I was involved with was a new range of programmable instrumentation, with remote operation via IEEE-488/GPIB and local front panel controls. This range included voltage and current sources, a programmable resistance, multi-way switch, and screwdriver *(1).  Later on, a multi-function calibrator was produced, combining several functions in one unit. Since I retired from actual paid employment a few years ago, I've been slowly collecting working versions of things I worked on during my career.  For example, using simh, I can run
Read more

DECnet-VAX Phase V P.S.I. configuration in open_simh

The addition of HDLC framing support to the DUP11/DPV11 devices in open_simh extends the number of networking products can be used under VMS in the simulator. The most obvious product to demonstrate is VAX P.S.I., which provides X.25 connectivity.  This example uses P.S.I. in its simplest possible mode, with two systems connected back to back, one configured to act as a DCE (X.25 is asymmetric).  P.S.I. is shipped as part of DECnet-Plus, so if you chose not to install it originally, re-run "PRODUCT INSTALL" for DECnet-Plus, and answer yes to the question about installing P.S.I. The rest of your configuration will remain unchanged. A valid license for "P.S.I." is needed to get started.   If you're feeling confident, you can just run @sys$manager:psi$configure and go through the configurator, to produce your own startup scripts. To avoid needing to run the configurator, and just use the example scripts below, do the following while logged in as SYSTEM: $ set def
Read more