Coisas CMS for Github Markdown

I just noticed the Coisas CMS (https://github.com/fiatjaf/coisas) for editing Markdown etc content on Github. So cool!

TODO: Link a lot of my prior work/research into this area

Questions

Recovering data from a failing Android adoptable storage

Background

My phone running Android stopped being able to read its Adoptable Storage™ and so lost all non-stock apps and their data (Riot, OSMand+, Signal, etc) as well as the shared “home dir” containing DCIM, Documents, Downloads. Most of this is backed up in various ways but I’d like to get the whole storage back.

The phone is rooted, using the LineageOS method with ADB root access enabled, so I can copy the encryption keys with a USB cable.

Solid state memory devices are sensitive to power supply issues, and I’ve long suspected my phone’s power supply is flakey since I often see spontaneous reboots and assorted random failures. My hope is that the card will be readable by a device with better power (it was!)

Plan

  1. Image the card
  2. Get the key from the phone using ADB
  3. Decrypt
  4. Copy image to a new encrypted card

Note: 1 and 4 can be completed without 2 and 3, although resizing the partition to the new card size wouldn’t be possible.

Image the failing memory card

I removed the SD card from the phone and inserted it into my Linux laptop’s SD port.

First rule of storage device recovery: create an image and work on the image.

Note: I’m creating an image of the entire device rather than just the encrypted partition. This will allow me to restore to a new card later.

First attempted imaging with dd, but saw I/O errors.

Attempt with dd

~/backups/herolte_recovery   
❯ sudo dd if=/dev/mmcblk0 of=herolte_sd_recovery_fulldisk.img status=progress
19326185+0 records in
19326184+0 records out
9895006208 bytes (9.9 GB, 9.2 GiB) copied, 208.766 s, 47.4 MB/s
dd: error reading '/dev/mmcblk0': Input/output error

Indeed,

❯ journalctl -b --since "10 minutes ago"
Mar 24 13:13:52 chip kernel: print_req_error: I/O error, dev mmcblk0, sector 19359824 flags 80700
Mar 24 13:13:52 chip kernel: print_req_error: I/O error, dev mmcblk0, sector 19359871 flags 80700
<30 lines omitted>
Mar 24 13:15:23 chip kernel: print_req_error: I/O error, dev mmcblk0, sector 22323535 flags 0
Mar 24 13:15:23 chip kernel: Buffer I/O error on dev mmcblk0, logical block 22323535, async page read

ddrescue

Time to try ddrescue, a time-tested utility for reading data from dying block devices. (archwiki)

❯ sudo ddrescue -f -n /dev/mmcblk0 herolte_sd_recovery_fulldisk.img herolte_sd_recovery_fulldisk.log
GNU ddrescue 1.23
Press Ctrl-C to interrupt
     ipos:   31673 MB, non-trimmed:        0 B,  current rate:   9092 kB/s
     opos:   31673 MB, non-scraped:        0 B,  average rate:  32025 kB/s
non-tried:        0 B,  bad-sector:        0 B,    error rate:       0 B/s
  rescued:   31673 MB,   bad areas:        0,        run time:     16m 28s
pct rescued:  100.00%, read errors:        0,  remaining time:         n/a
                              time since last successful read:         n/a
Finished                                     

That went considerably better, no IO errors at all. Why the difference from dd? ddrescue is optimized in various ways to deal with unreliable block devices.

Fetch key from phone

@kapitanpetko’s post from 2015 is the original source all the related stack exchange articles seem to draw from (usually without reference).

Allow reading naughty bits (requires phone rooted for adb access):

❯ adb root       
* daemon not running; starting now at tcp:5037
* daemon started successfully
restarting adbd as root

Fetch the key:

❯ adb pull /data/misc/vold
/data/misc/vold/: 1 file pulled. 0.0 MB/s (16 bytes in 0.013s)
❯ ls vold
bench  expand_ffffffffffffffffffffffffffffffff.key
# convert bin to hex
❯ hexdump -e '16/1 "%02x" "\n"' vold/expand_ffffffffffffffffffffffffffffffff.key
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa%         

(source for plain hexdump format string)

Mount encrypted image

First map the image to a loop device so dmsetup can use it. Note that the -P flag creates a device for each partition in the image.

❯ sudo losetup --show --find -P herolte_sd_recovery_fulldisk.img 
/dev/loop0

Map the encrypted device:

chip# dmsetup create crypt1 --table "0 `blockdev --getsize /dev/loop0p2` crypt aes-cbc-essiv:sha256 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 0 /dev/loop0p2 0"

Mount filesystem:

❯ mkdir mnt
❯ sudo mount /dev/mapper/crypt1 mnt
❯ ls mnt 
app  local  lost+found  media  misc  user  user_de

Looks good!

Copy to new card

  1. Image the physical block device (not just the encrypted partition) and write that to a new card, and hope Android doesn’t distinguish between physical media. This also depends on the new media being at least as large as the old one.
  2. Use the phone to initialize a new card, mount it on the laptop and copy files over

Side note: Option 2 was a dead end because Android didn’t see any of the apps.

Attempt to copy files to new encrypted partition

The 32GB replacement cards I ordered are actually physically smaller (28.5G vs 29.5G) than the original so I wouldn’t be able to restore a full device image to one of them. I have a 64G card that I could copy to the image to but that would likely result in half the card being unused, so I opted to try copying the data to a new encrypted partion.

  1. Insert the new 32G card in phone and format as encrypted.
  2. Get key as above.
  3. Mount the card as above, but skipping the losetup step and using mmcblk0p2 instead of loop0.
  4. With both the loop0 image mounted and the new card’s physical partition, remove contents of new partition and copy data over with rsync -a srcmnt/ destmnt.

Unfortunately as noted above, android doesn’t recognized apps copied over this way.

Copy full device image

Plan:

  1. Image the entire card rather than just the encrypted partition.
  2. dd the image to new card.
  3. Grow the encrypted partition, since the new card is 64G while the old was 32G.

Ok, now what I’d like to do is copy the entire filesystem to a new card. My new card is 2x the size of the old card, so I’ll grow the partition after copying. If your card is smaller you’ll need to shrink the partition before copying.

❯ sudo dd of=/dev/mmcblk0 if=herolte_sd_recovery_fulldisk.img bs=4M status=progress
[sudo] password for eric: 
7555+1 records in
7555+1 records out
31691112448 bytes (32 GB, 30 GiB) copied, 729.118 s, 43.5 MB/s

At this point I ran sync and put the card back in the phone. It works!

The only problem is, because I copied a 32G image to a 64G card, only half the available space is usable.

Resizing the encrypted partition

Normally growing a partition is a two step process:

  1. Grow the partition
  2. Grow the contained FS

(for shrinking you reverse the steps)

This turns out to work mostly the same for an encrypted partition.

Resize partition

❯ sudo parted /dev/mmcblk0
GNU Parted 3.2
Using /dev/mmcblk0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) pr                                                               
Warning: Not all of the space available to /dev/mmcblk0 appears to be used, you can fix the GPT to use all
of the space (an extra 63145984 blocks) or continue with the current setting? 
Fix/Ignore? fix                                                           
Model: SD GC2QT (sd/mmc)
Disk /dev/mmcblk0: 64.0GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name            Flags
 1      1049kB  17.8MB  16.8MB  fat32        android_meta
 2      17.8MB  31.7GB  31.7GB               android_expand

(parted) resizepart 2 100%
(parted) pr

Number  Start   End     Size    File system  Name            Flags
 1      1049kB  17.8MB  16.8MB  fat32        android_meta
 2      17.8MB  64.0GB  64.0GB               android_expand

(parted) q                                                                
Information: You may need to update /etc/fstab.

Resize filesystem

First map the encrypted partition using the new size:

dmsetup create crypt6 --table "0 `blockdev --getsize /dev/mmcblk0p2` crypt aes-cbc-essiv:sha256 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 0 /dev/mmcblk0p2 0"

Now resize the filesystem:

❯ sudo resize2fs /dev/mapper/crypt6
resize2fs 1.45.0 (6-Mar-2019)
Please run 'e2fsck -f /dev/mapper/crypt6' first.

❯ sudo e2fsck -f /dev/mapper/crypt6
e2fsck 1.45.0 (6-Mar-2019)
Pass 1: Checking inodes, blocks, and sizes
Pass 1E: Optimizing extent trees
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/mapper/crypt6: ***** FILE SYSTEM WAS MODIFIED *****
/dev/mapper/crypt6: 47391/1933312 files (1.0% non-contiguous), 3824859/7732731 blocks

❯ sudo resize2fs /dev/mapper/crypt6             
resize2fs 1.45.0 (6-Mar-2019)
Resizing the filesystem on /dev/mapper/crypt6 to 15625979 (4k) blocks.
The filesystem on /dev/mapper/crypt6 is now 15625979 (4k) blocks long.

Wrapping up

❯ sudo dmsetup remove crypt6
❯ sync

Your new card should now show full capacity in android.

Quirks

For some reason in my case Android shows a fictitious extra 64G of used space and capacity, maybe because it’s confused by the device having different metadata.

Screenshot of storage settings page

2015-03-09

Sparklines

2014-06-23-Open-Source-Bridge

Paul Fenwick - Build your own exobrain

bee minder track things

Bart Massey - Random

Mobile Accessibility

FirefoxOS

http://www.w3.org/WAI/IndieUI/ http://www.w3.org/WAI/intro/aria

Citizen Science

https://etherpad.mozilla.org/openscience https://en.wikibooks.org/wiki/Citizen_Science

Kronda Stop Crying in the Bathroom and Start Your Own Business

Coworking spaces in Portland, OR

Three collaborative spaces I know well enough to recommend, written up in response to a friend’s search.

Collective Agency

Open floorplan collectivist space, with lots of community facilities including a full kitchen and side rooms with whiteboards and phones. Some have reported eerie quiet in the main area during the day weirds them out but you should try it and see for yourself.

They host an all-day open house once monthly.

Hatch

A mix of open plan and “fireboxes” which are little cubes (what a bad name right?). Less techie, more traditional professional services vibe. Their “5 day a month” membership level includes access to conference rooms. Also in a cool neighborhood, which is nice for thinking/walks.

Nedspace

A Portland institution and that has helped filled the void (along with CoAg) left when Cube Space closed in 2009.

FUSE vs the Temp Files

I moved my Documents folder into camli (on my home ARM server) and have been accessing it via FUSE for 10 days or so.

Overall, the FUSE filesystem has been unstable.

1. FUSE exits occasionally without explaination

This is probably an easy fix, I haven’t looked at the code yet.

2. writes block until everything has been uploaded successfully

This is painful over slow links. Not only does the writing app (VIM/sublime) block, but also things tmux (not sure why, maybe only one thread in FUSE somewhere?)

3. it doesn’t provide a way to exclude files (such as temp files) and metadata operations (xattrs), so lots of garbage gets created

Support for excluding specific files/operations (xattrs) would require a local buffer of some sort, either a blob store or a real filesystem overlay, which could be flushed when FUSE is unmounted.

One short term bit which would help is a switch to disable xattr support at the FS level.

Also, tell apps to write their temp files elsewhere and disable atomic writes;

camli personal storage system start server cammount import everything similarity to git recent importer rewrite (web config ui, store api creds and user auth tokens as data objects) importer rewrite foursquare

Beaglebone Storage Server

install debian

My Beaglebone (shipped March 15) came stock with Angstrom, which I replaced with the official BB Debian image. Beaglebones are now shipping with Debian preinstalled.

trim down

The Beaglebone Debian image comes with a bunch of useful stuff (Git, tmux) but it only leaves about 120MB for installing application packages, log files, etc.

To see a list of manually installed packages sorted by size, open aptitude and

list of possibly unnecessary packages:

secure login

hdd reformat

home network

My home router is DD-WRT. I highly recommend using it over stock vendor firmwares.

public network

cross-compile camlistore (on laptop)

camlistored startup

syncc

Upcoming

2013-12-06

Grid systems research

https://wiki.thingsandstuff.org/Grid/rhythm

http://fourkitchens.com/blog/2013/10/14/singularity-grids-without-limits

Prioritize result over process purity

I am using Harp to build my pages. However, it is sub-optimal when it comes to CSS and JS builds.

Maybe something like this makes sense:

Unfortunately, Harp doesn’t support –watch for compile mode. This needs to be fixed I think.

Color

An ICC profiles that “fixes” CIELAB (called Uniform Perceptual Lab) http://www.brucelindbloom.com/index.html?UPLab.html

http://www.littlecms.com/

Fonts

2013-11-27

Thinking about:

Build my site service

Sub-document granularity

TOYS!!

What did I get out of brainstorming today?

2013-11-22

Ok, let’s try yesterday’s prescription.

Feeling: like a storm has passed woke up to birthday wishes (i forgot it was my birthday)

Yesterday I rode my bike a bit, but still didn’t get enough exercise I think since I felt weird in the evening.