none of this matters

Lesson #1 of embedded development: don’t write all your code to your device and not backup/check-in. I did just that when writing some code for the hydroponics monitoring setup on my Fon2100 router running OpenWRT. After trying to opkg install ntpclient and getting a segfault and NTP not working, I restarted  the network on it which restarted the router. For whatever reason, the router choked and each restart went into failsafe mode. When I telnet’d into it, dmesg told me the jffs2 partition had trouble mounting with an error in jffs2_link_node_ref.

I had written a bunch of Lua code to read the serial communications coming from the ATMega168 microcontroller (it was sending its analog input values) and store it into files in /tmp which were then read by some more Lua code in the (awesome) LuCI framework to make those sensors all cool and AJAXy on the router’s web-admin menu. This worked great until the NTP attempt. So I wanted all this code back. So in case anyone is as foolish as I was to lose their code in the router, here are the steps I took to recover it. David Woodhouse (dwmw2) and dedekind at #mtd on irc.ipv6.oftc.net were extremely helpful with all this and pointed me in all the right directions.

1. backup the jffs2 image from the router using dd. mine was coming up in failsafe mode so i had to telnet into 192.168.1.1 and then run command `dd if=/dev/mtdblock2 | gzip -c | ssh nemik@pillbox ‘dd of=/tmp/mtd2.gz’ bs=2048` but make sure you do the right mtdblock2 device. do `cat /proc/mtd` and choose the one that is for “rootfs_data”. pillbox is a Linux box on my network I transferred this all to.
2. `gunzip /tmp/mtd2.gz` on the Linux box and i got my 5.5MB partition.
3. wget ftp://ftp.infradead.org/pub/mtd-utils/mtd-utils-1.2.0.tar.bz2. untar, make, make install
4. `sudo mknod /dev/mtd0 c 90 0`
5. `sudo modprobe mtdblock`
6. `sudo dd if=/tmp/mtd2 of=/dev/mtd0 bs=2048`
7. `sudo mount -t jffs2 mtd0 /tmp/1/` THIS DIDN’T WORK! (right away). The Fon router and openWRT are big-endian and I was trying to mount this on a little-endian x86 box. I needed to recompile the jffs2 kernel module to be big-endian. David Woodhouse (creator of JFFS2) writes how here: [http://www.infradead.org/pipermail/linux-mtd/2007-May/018227.html] I needed to get the kernel source and recompile the jffs2 module
8. get the kernel source `apt-get source linux-image-$(uname -r)`
9. go into the source and do `make prepare_modules`
10. copy /usr/src/linux-headers-2.6.27-14-generic/Module.symvers to the source. this will be different for you, but the Module.symvers is important otherwise the module won’t load properly
11. Change the native_endian #define in fs/jffs2/nodelist.h to be little_endian as David writes in the infraread link on step 7.
12. do `make CONFIG_MODVERSIONS=y M=fs/jffs2`
13. if it compiled correctly, you then have a jffs2.ko module. load it via `sudo insmod /home/nemik/code/jffs2/linux-2.6.27/fs/jffs2/jffs2.ko`
14. now try `sudo mount -t jffs2 /dev/mtdblock0 /tmp/1/` and it worked for me!

I was now able to see all my lost files in /tmp/1/ !!! the whole filesystem was there and worked great. I extracted all the code I wrote and learned my lesson never to do development on a router and always backup and check in. Hopefully this helps someone out.