Apps Linux Mac Networking Other

Rescuing Encrypted files on ACD

So Amazon is shutting out Linux users.  But what if I have a bunch of encrypted files there using old encfs and acd_cli scripts?

I can copy down the encrypted files using their client at any point, but how will I know which one is which?

I did the following.  First, create a temporary directory.  I did this in my $HOME on my Mac.  Find a way that still exists to mount the drive (I used ExpanDrive).  Once that is prepared, change to the mounted and encrypted ACD directory and run this command:

find . \( -type d -exec mkdir -p "$HOME/temp/{}" \; -o -type f -exec touch "$HOME/temp/{}" \; \)

Let it run for a while, it may take several minutes.  This will create in $HOME/temp the identical directory structure as on the remote drive, and the identical filenames – but they will all be zero bytes!  What good is this?

Thanks to the consistency of encfs, you can mount and decrypt this skeleton directory like this:

ENCFS6_CONFIG=/path/to/your/encfs6.xml encfs $HOME/temp $HOME/plain

Now, use some other tricks to find the matching filenames and you can manually download the specific encrypted files you want.

Linux Networking Other

Amazon Drive Shutouts

Amazon is in the process of revoking API keys for a number of apps used to access their Cloud Drive product.  They started with acd_cli and have now banned rclone, and apparently Stablebit CloudDrive.  As more users of these products filter into fewer and fewer available options, the API hit from those products will certainly increase, possibly leading to more bans.


Amazon is theoretically overselling cloud drive capacity.  They call their service “Unlimited” but are counting on the fact that no sane person will upload more than a few hundred gigabytes of data.  Using their client, yes, that would be true.  (To be sure, their client has improved considerably in the last 6 months or so).  The above clients, though, allow you to mount your remote drive like any regular network-attached drive, and work directly off it.  With that ability, it is worth putting terabytes of files on a cloud drive and freeing up local resources.  Some files, like video files, are ideal for this application.

Some very sane people started storing 5, 10, 50 and more terabytes on their service.  Can you imagine the accountants panicking?

But I said “theoretically” overselling.  Is that true?  Are these cloud drives really losing money by allowing large amounts of storage?  Let’s calculate the cost of storage, in a grossly oversimplified calculation.

I looked up the price of a 2 TB hard drive this morning.  It was on sale for $89.  If I wanted 2 TB of cloud storage for one year, at that price it should cost ($89/12) $7.42 per month.  Do you know any cloud services that offer that price?   Amazon’s Cloud Drive is $60/year for “unlimited” space.  Based on the price of this drive (and I know this bears almost no parallel with reality), they are expecting to split the purchase cost of this drive between ($89/5) 17.8 people, who each store a maximum of (2000/17.8) 112.3 GB each.  Would you pay $5 a month to store only 112 GB of data?  Me neither.  Amazon naturally doesn’t expect that either, so it likely means their cost is far, far cheaper than that.

Where’s the “value-line” for you at that $5/month price though?  500GB?  1TB?  2TB?  I’m guessing it’s somewhere around the 1TB mark, with options for a little more.  (Incidentally, that’s kinda the price of Office365, including 1TB of storage and, oh yes, a full office suite thrown in for “free”).  Would you pay, oh, $7.42 a month for that capacity?  Wait, that is totally coincidentally the price of a 2 TB hard drive!  And you get to keep it after a year, get a second one for replacement or expansion, every. single. year.

Their business is NOT based around cheap, small hard drives at retail prices.  They run a staggeringly huge data warehouse with storage pools.  Their pricing is a fraction of what ours are, allowing them to pay for the rest of the overhead with attaching that drive to the Internet.  They can likely afford a few hundred outliers.

So it’s not just about storage space.  It’s about reliability, availability, and convenience.  Kinda those things you lose when you shut out third party apps eh, Amazon?

So if I can’t effectively store my video backups up there, and if they close off more of their API so I can’t use Arq for backups… my practical use for ACD drops way, way down below my “value-line”.  It may be time to migrate.

Linux Networking Programming Server

Managing Encrypted files on Amazon Cloud Drive

I have implemented a file system on Amazon Cloud Drive for a lot of media with the great acd_cli.  To protect my privacy, I have run this through an encryption layer encfs.  My writeup will follow.

A problem I was trying to solve in my mind though, is how to manage – rename and delete files once they’re all scrambled up and I can’t discover even the path and filenames.

Ultimately this would be seamless.  Delete a local file stub and it traces back to the encrypted remote file, but it doesn’t quite work that way.  I discovered how to do this on my Linux host.

Once I realized that the filesystem for encfs has the same inode numbers for the encrypted and decrypted files, I had a clue.  First, let’s find out what that file number is:

$ ls -li cloud.plaintext/subfolder/filename.ext

149 -rwxrwx--- 1 jonathan plex 597979891 Dec 27 05:14 cloud.plaintext/subfolder/filename.ext

149 is the part we want.  inode numbers are unique per partition/filesystem, and seems to persist between the encfs pairs.  Now, to find a file in the encrypted path system with inode 149… find to the rescue!

$ find cloud.encrypted -inum 149

cloud.encrypted/(encrypted subdirectory name)/(encrypted filename)

I won’t even try to copy/obfuscate the number above.  Try it if you want to see it.  It would be almost impossible to track that file without the number.  Size and date are much harder to nail down the exact file.

So, to stitch these two together first you want the inode number only:

$ ls -li cloud.plaintext/subfolder/filename.ext | cut -f1 -d' ' 149

Now this is something we can use in a delicious Linux command chain.

$ find cloud.encrypted -inum $(ls -li cloud.plaintext/subfolder/filename.ext | cut -f1 -d' ')

cloud.encrypted/(encrypted subdirectory name)/(encrypted filename)

This is easy enough to make into a little bash script, and allow passing arguments and quoting to protect against embedded spaces, as well as including the explicit Amazon Cloud Drive working area:


find ${ACD_LOCAL}cloud.encrypted -inum $(ls -li ${ACD_LOCAL}cloud.plaintext"${1}" | cut -f1 -d' ')

Works great for specific files, not so much for directories.  You would have to change the ls command to use a -ldi parameter just for those cases.

Now that we have the filename, we can manually delete that filename on Amazon, either through the web interface or using acd_cli’s command line trash argument.

Music Other

A Dip in the Music Stream

Music streaming has become a big deal in the last few years.  People don’t seem to care about bandwidth, they know its a resource that seems to be basically infinite (if metered).

My problem is that it almost seems like the users are treating music the same way.

I reflect back to my discovery of music.  It started off as a social thing – you wanted to hear the new stuff that other people were talking about.  These were the days of carrying a boombox around so you could play your tunes for everyone who surely would like it as much as you, right?

After not long though, I realized there were things that others liked that I didn’t.  Why?  If it was popular, wasn’t it automatically good?  I realized that music can be something very personal, you can find some lyrics or tones that speak to your soul, but not those of the people right around you.  And, you know what, that cut both ways.  Maybe music I really enjoyed was never going to be the favourite of others [1. I know, seriously?].  I think I am still learning this, actually.

This is not to say that music lost its social aspect, sometimes it defined it.  This may have been an operation of the teen years as well, but there were times I selected or prioritized friends based on their music tastes.  I want to reject this as immature, but perhaps there is some small bit of validity to this…

Nevertheless, over this time, I began curating my own collection.  Thanks to Columbia House and BMG I gained a stack of CDs (vinyl was from record stores).  Much of it was familiar but I could afford to experiment a bit.

Music was tangible.  Music had a measurable value.  When I got a physical CD, it was an investment.  Sometimes I didn’t really like it on first listen, but since I paid for it, and I only had so many of them, I’d better get into it.  Sometimes I loved it from the first listen – yet I would usually limit myself to one listening a day, so I didn’t get tired of it.  Leaving it running all the time was absurd and wasteful.

To me, digital music downloads were the same thing, the lack of a physical disc didn’t cause a disconnect between the music and “some thing of value”.

This finally leads back to streaming.  I have tried a few services.  I ran Rdio for a long time, it was quite good, and I discovered some new artists I really enjoyed.  Then I tried Spotify for a month, it was also good, and I occasionally used some of the stations.  I ended up getting Xbox Music Pass for a year.  I tried Apple Music, and Google Play Music.  Back to Spotify for a few months.

But sometimes when I listen to stuff on any of these streaming services, I can’t shake the feeling that lots of this music isn’t worth much, and that has absolutely nothing to do with the compositions or the performance.  It seems very temporary, and that bothers me a little.  Why am I wasting my precious listening time on “throwaway” music?

I’m slowly coming around to the value of streaming.  I have discovered a few outstanding bands and songs that are valuable to me and exist only in that context.  But like air and clean water, maybe it’s something we have to stop ourselves and think about their value.

In the meantime, I’m gonna dial up some Big Country and Flock of Seagulls.  Enjoy, my public.

Apple Apps iPad iPhone

The Best eBook Reader, Again

I came to the realization a few years ago that the eBook reader Marvin was hands down, the best ePub reader for me.  Very flexible, configurable and some (still) unique features made it well worth the price.

That was, as I mentioned, a few years ago and I started to get a little frustrated with a few important (to me) missing features, such as ePub 3 features like aside (pop-up footnote) support, syncing and a few other small things.  The lack of these was one thing that bothered me.  The fact that the same developer released a second, free ePub reader app that offered these features was even worse, and quite frustrating to those that paid for the original app.

It wasn’t simply an easy task to switch from Marvin to Gerty though.  It wasn’t designed as a generic ePub reader, but rather a sort of book-journaling app.  What was the developer thinking?

It turns out he was thinking.  And working hard.  He was rewriting Marvin from scratch.  Today, he released Marvin 3.  This release adds, and far surpasses my original wish list.  Now there is full iOS9 support, including document picker, spotlight integration and split-screen support.  The icing on the latter part of that particular cake is that there is a second, nearly identical app, called Marvin SxS (“Side by Side”) that lets you have two copies of Marvin installed on the same device and you can have two ePubs open at the same time in devices that support split screen.

Marvin on an iPad
Marvin on an iPad

There are lots of great built-in fonts, whether you are a sans or a serif fan.  There is also OpenDyslexic built in.  If you aren’t satisfied with any of the bundled (and system) fonts, you can sideload others.

Margin size, line spacing, paragraph spacing, indent size are all configurable.  There are themes.  Multiple columns available in both landscape and portrait.  Textshots and auto-bookmarking on close.  Reading location syncs automatically to iCloud (zero-config).  And then there’s annotations – all those things that were in Gerty are now in Marvin 3.  You can of course highlight with a load of colors, but also add notes and photos to a book.   Apparently there are multiple map-viewing modes (you can read maps in here?)

There is native comic book (CBR and CBZ) support, and it’s really, really good.  I have, and love, Chunky Comic Reader, but in practice, there are only a few significant advantages Marvin 3 lacks – PDF comics, landscape for dual-pages and ComicStreamer support.  The page thumbnails (with long-tap to preview the page) are wonderful.  The zig-zag mode is much like Chunky’s “pan” mode.  You can scrub through the comic and have page previews show you where you are.  Given that Chunky is iPad only, I might find myself using Marvin for some of my comic reading, depending on the amount of dual-pages I might expect.

I have only touched the surface.  The only other reader that comes close regarding configurability is Moon+ Reader Pro on Android.

It is not a free upgrade.  There are two in-app-purchases to unlock full screen use and color themes, $3.99 USD for the former and a variety of “tips” for the latter, although you only get certain themes depending on the amount you “tip”. Kinda sounds to me like a purchase and not a tip… The SxS version is a full purchase without IAP unlocks – other than the themes.  To me, the regular IAP (or outright purchase) is well worth the cost.  The colours, not so much, but that is merely an opinion.

The bottom line is, if you’re looking for the best ePub reader, get Marvin 3.


Steam Controller

The visible and eminently tactile part of the Steam hardware experience is the controller. Steam took a leap and designed something significantly different from the standard twin stick/d-pad “standard”.
The fundamental reason for doing this is that Valve is designing a controller for their platform and not for a subset of games that are already suited for controllers. This is a tall order, and needs to take in FPS games, adventure games, strategy, racing, and everything in between. And it kinda almost works.

I tested a few games to see how it worked.

It’s easiest, though perhaps unfair, to compare to the well known (and loved) Xbox 360 controller.

The Steam Controller has an overall totally new tilt. It is designed so that your thumbs are in place to float over the touchpads, instead of the A/B/X/Y buttons. The shape is more concave than convex, like virtually all other gamepads. The grips of the controller sit nestled in towards the palm/heel of your hand.

Out of the box, the controller feels rather light, when you add the batteries it feels better, more balanced.  It is still lighter than most gamepads, but probably partly because of the different style of haptic feedback,rather than weighted “rumble” motors.

Since the touchpad is central to the operation, the buttons have been bumped down and to the left. Unfortunately this requires a reach, instead of them being instantly available, and locatable by touch. Another side-effect is that the bottom left edge of the touchpad is a dead zone to prevent hitting there when you use the B or Y buttons.

There is no D-pad at all, though the left side can work as one. Since it is a touchpad, though, you can slide up and down (for things like mouse wheel actions), amongst it’s many configurable layouts. This offers a lot more ability, if a little finicky. Both touch pads can be clicked.

The trigger buttons are analog with two “steps” of pull. You can half-pull them for one action and pull all the way to perform another (there is a slight click). There are digital shoulder buttons above that, and a wonderful “squeeze” button under your middle and ring fingers.

Above all of that, there is apparently a gyro inside (motion/tilt tracking, not the delicious Greek food). To wrap all of that up, there is naturally a clickable analog left thumb stick, as well as a few ancillary buttons.

The most powerful part is the software. Without any third-party add-on drivers or utilities you can make extremely fine customizations to the functions of every control, from mouse-vs-joystick type controls, to haptic feedback, to sensitivity. You can configure an outer ring for each trackpad to a different function. You can use a key as a mode switch, to double the number of commands. It is certainly possible to program the controller to handle all but the most complicated and confusing keyboard and mouse scenarios.

You can set up a configuration for each game (and download community-created ones too), and one “Desktop configuration” that can be used to make the controller a general purpose input device, so you can play games without running through Steam. That being said, I haven’t figured out a suitable Minecraft setup yet, and yes that makes me sad.

With that ability comes the caveat. You will need to customize it. You will need to spend time playing with the settings instead of playing with the games. By now, thanks to the preorder and early release, there are the above-mentioned community-provided profiles for pretty much any game. It’s easy to pick one that looks reasonably familiar and tweak it from there.

Nevertheless, it is a totally new design of a controller. You can only pick it up and be instantly familiar with it for some games here and there, maybe like some racing games, platformers and so on, but anything else needs you to relearn what a controller feels like and can do. It will take some time. F1 2013 was instantly comfortable, Portal Stories: Mel took a little getting used to, some various RTS games were and still are a big challenge for me.  Tomb Raider Anniversary was tremendous.

I think it has a lot of promise, and I am still not totally comfortable with it. Valve can easily influence game authors and publishers to build controller support in to the games, and in fact they may already have done so.

We as gamers are not totally strangers to innovation, but maybe we usually expect it from Nintendo. PC gamers are perhaps more locked in to their keyboard and mouse mindset, but this controller is definitely not designed to completely displace the desktop input devices.  Will it displace traditional gamepads?  Time will tell.  Perhaps version 2 or 3 will be even more devastating to them.

Bottom line is, there’s a entirely new niche of game controllers that has been created, and it is currently occupied solely by the Steam controller.  It can control all of your Steam games, and that’s a pretty impressive resume.

Apple Apps Electronics Games

AppleTV Updated

Finally, Apple decided to update the AppleTV – and promote it to the product menu bar on the main site!  This common, and popular, device had been overlooked for four years!

The changes are exactly what I was wishing for… but somehow not enough.  The main thing I wanted was apps.  It seemed kinda dumb that an iOS device (the previous AppleTV was running iOS 5) couldn’t run apps.

The internals have been improved significantly to support the load that modern apps will certainly give to the hardware.  An A8 processor (mostly the same in the iPad Air 2, with a different GPU), 2GB RAM, and 32-64GB flash storage.  Bluetooth 4.0 and WiFi 802.11 ac (along with the other letters).

But Apple has never been strictly about the internals.  You shouldn’t need to know exact RAM numbers to know how the experience will be [1. Quick, tell me how much RAM a PS3 has!  Whether you know that number or not, you realize it doesn’t matter as much as it would on a desktop].  Apple has always been about the experience, even if it causes the device to be a little more expensive.

So what is the experience?  Well, the existing AppleTV experience is a gimme.  It was already very good and uniquely seamless… playing iTunes content and receiving AirPlay streams is a very satisfying experience.  The downside?  The doors were closed to anything else.  So now with apps, the experience gets much wider.  Plex is better than iTunes.  [2. There.  I said it.]  TuneIn Radio would be nice to have on the same box.  And then there’s games!

<record scratch>. [3. For the younger set, you might have to Google this]

Of course there’s going to be games, when people talk about the App Store for iOS, their first thought is GAMES.  But, the App Store games have been unique beasts until now.  Odd games with ridiculously simple controls and brief (seconds to minutes) of typical game time.  Games like Threes, Angry Birds and Plants Vs. Zombies really worked on a portable, touchscreen, battery-powered handheld that you had with you anyway.  Put it on a screen with three other bored family members watching and it won’t work.

The games have to change nature, obviously.  When you fire up a game in the living room, you do so with hours, not minutes in mind.  Solo games are still fine, of course (they already succeed on the consoles, so that proves that).  But the AppleTV as a gaming device is going into the arena with a handful of potential games worth playing.  I too loved the demo of Crossy Road, it looked great… but I think I would only play it on the big screen for 10 minutes.  The Apple ad highlighted Asphalt 8, and there’s also Real Racing 3, another graphics-beautiful game that lends itself wonderfully to the controller.  And with the In-App Purchases you can…

<record scratch>.  <audience gasp>. <crickets>. [4. You have to add your own sound effects here.  Work with me, people.]

I bought Real Racing 2.  It was fun, if a bit unrealistic.  It was self-contained and cost money up front.  Real Racing 3 came out, and was mostly improved except it’s now Freemium… so you can play for free as long as you stop playing every few races to let it recharge.  Stop.  Playing.  Tell me if that’s going to work when you’re on the couch with a cool drink and controller in hand.  I know, there are about 1% of the players that actually pay for cubits or gemoids or coinules or whatever they have [5. For the older set, don’t bother Googling these].

The easy(tm) solution is to charge more for the games.  I’m actually fine with that, as I’d pay $5-$10 for a really good game.  That would be kinda the upper limit, the reason for which will I will explain shortly.  So maybe that will happen.  I’m actually pretty certain(tm) that it will, actually, though I think the prices will balance out at a little more around $10-$20.

Now back to the hardware.  It has the grunt to drive these games, but there are two significant problems that stand out right away.

First, controller support is restricted to MFi controllers, so those awesome and cheap USB controllers you already have (like the ubiquitous Xbox 360 controller) cannot be used.  So you can get an AppleTV for $149 plus a controller for $49 more.  $200 USD for an unknown console with a single controller… now that’s not too bad given new console prices.  Except I could pop out today and get a PS3 with 4 really good games (Lego Batman 3 and the Sly Collection) for $219 CDN.  With 500GB of storage.

Which leads me to my next issue.  There are only two levels of storage differentiating the $149 and $199 models Is 32GB enough, or is 64GB enough?  Right now very few know.  A high-end iOS game at the moment might push 2GB, or possibly more, but recently released developer information explains that apps for the AppleTV can only reserve 200 megabytes of persistent storage each.  They can download and cache a whole lot more than that, apparently, but given memory pressure (i.e. lots of different apps, not just games) using that cache, it will be dumped and you will have to redownload it.  So the snappy performance of the device when you first set it up will probably fade.  In practice, maybe you don’t notice this much, since you’d have to run a lot of different apps to create the memory pressure required to dump the cache of one.  Most people would focus around 4 or 5 apps each.  Unless of course you’re a family.

Oh… yeah, that [5. Won’t someone think of the poor RECORDS?].

It would be easy to compare this to the Xbox One, because it does much of what the Apple TV promises, without the Apple integration, of course… but it’s expensive, so I won’t do that.

Let’s go back briefly to the pricing of games…  It seems to me the big competitor they’ve just walked into the ring with is Valve.  Steam sales are ridiculous.  It’s not unusual to get a significant and excellent game on Steam for under $5.  Steam games are often cheaper than their iOS ports.  But, PC games and consoles are very different things.  The other key is the Steam Link hardware that is coming out next month.  A $59 CDN box will stream your PC/Mac games to your big screen.  It supports Xbox 360 controllers as well as a few others, and they are releasing a new $59 (CDN) controller as well, that is looking promising.  So for games, it’s $119 for a game plus controller, and a TON of cheap, high-quality games.  And an online store that works really well.  It will also stream your music if you really need to (MP3 only).  Maybe movies will come soon, who knows.  While it’s true it’s a streaming-only device, and you need your computer on… AppleTV is mostly the same for movies and music.  (Yes I know about iCloud)

I think the real potential competitor to the new AppleTV is the old AppleTV + Steam Link.  $89 + $59 = $148 CDN beats $149 US by a fair margin.  Enough to throw in a controller, if you really really don’t have one.

Comparing unreleased, unknown products is often an exercise in madness.  But at least my madness is well exercised.  I think we’ll know how this all shakes out in about a year.  Mark this down.


My Keyboard Journey

As a professional developer, I spend a lot of time with my hands on a keyboard.

Many many years ago, I was privileged to use an IBM Model M keyboard.  I really enjoyed it, but trends changed.  Microsoft came out with a fancy looking ergonomic keyboard and I had the chance to try it out.  I liked the shape, but the keys – especially the space bar – were rather unpleasant.  So, I went back to the Model M.  I even managed to pick up a “space saver” TKL (“ten key” less) version, which is designed more for server racks

MS ergonomic keyboard
MS Natural Ergonomic 4000

Over time, though, I just used whatever was attached.  My jobs at the time just had a bundled Compaq or Dell keyboard and I just used it.  Unless they keyboard was really terrible, I managed and didn’t make a fuss.  When I started working for myself, I tried out a Microsoft Natural Ergonomic 4000.  They had come a LONG way since the first one, the key feel was much better.

After a little while, I switched to a Mac.  The Apple keyboards are actually pretty good, better than most, but they just don’t have much key travel, this is the nature of their low-profile design.  I plugged in my Microsoft Ergonomic again.  I decided I liked it enough to even get a second one after the first one wore out.

IBM Model M
Two words: awesome.

All of this time, I still had my Model M.  It stayed with me through about 5 moves, stuck in a closet, still has the exceptionally long PS/2 cable (still coiled up, it’s probably 10′ long).  It’s heavy, it’s fairly loud.  It uses “buckling springs” under the keys that gives a unique progressive resistance and feedback.

I never had any problems with my hands and wrists with this keyboard.  I figured it must give your hands enough strength to keep healthy, or something like that.  But, it was PS/2.  So it limited my opportunities to use it.

I started wondering if I could get a better keyboard, a mechanical keyboard rather than a membrane keyboard.  This is the only way to get really good feedback from each key.  Unfortunately, cheap keyboards are so popular because they’re, you know, cheap.  A good keyboard starts to cost something.  I thought about it though, and realized I would pay a fair bit for a really good mouse but for some reason always cheaped out on a keyboard that I use much more… so… I set myself a slightly higher budget and started to search.

It seems like most of the focus these days for these higher-end keyboards is on gamers.  One main feature is called n-key rollover, so you can hit a whole bunch of keys and have them all register.  (Here’s a test to try your own keyboard: hold down both shift keys and try to type “the quick brown fox jumps over the lazy dog”.  Yes, this is challenging.  If you don’t have a similar feature, you may be dismayed to see many letters missing).

There are a number of flavours of key switches, mostly from the Cherry company, they are identified by a color, such as black, blue, red, etc.  It’s very hard to know what kind you want, because very rarely are these keyboards available to try in stores!  I managed to find one or two, and gave it the swat test, and did not like it at all.  (It seems like the one I tried were the blue variety).

Other features are cool, but not that necessary for work, like LED backlighting.  I didn’t need to see this feature, though it would be cool.

My new keyboard
My new keyboard

I finally found in a review a reference to the Matias tactile keyboards.  I checked them out and it sounded really good.  I wasn’t worried too much about the volume, though I heard the Tactile Pro described as “antisocially loud”.  They also made one called the Quiet Pro, though, and I thought that only offered advantages over the non-quiet version.  They had a Mac version!  If the caption didn’t give it away, I got it, and I’m using it right now.  It’s really really good!  It’s really easy to get used to typing on one of these, so that when you go back to a cheap keyboard it feels jarringly wrong.

At the same time, though, I found a wonderful project online called something like TMK which is an open source driver/converter for older keyboards.  With Teensy (or a few flavours of Arduino) you can convert a PS/2 keyboard to USB and add a ton of really cool features to it at the same time.

What this means is… I realized my Model M can come back to life!  I have most of the parts and at least one more on order, but I will build this very soon.  I will report when I get it running.  I will have my choice of great mechanical keyboards, well, at my fingertips.



The recent update of iOS and OS X brought along Apple’s new cash grab photo organizing scheme framework.  To be honest though, it is very slick.  It makes photo storage and access very secure and ubiquitous.  You don’t need to worry about where your photos are and you (probably) don’t need to worry about having space on your devices.  Like Apple’s typical catchphrase, “it just works”.  It kinda represents the ideal of cloud storage – you don’t know or care where it is, but it follows you everywhere and rains your data down whenever you want it.  Also it obscures the sun.  And makes visibility at higher altitudes a bit of a challenge.

Hold on, those last two might not fit in to the desired description.  Just leave them off for now.

Except this convenience is gonna cost you.  iCloud storage prices have come down, but they are still a little on the pricy side.  The lowest end (20GB) is $1 a month, and perfect for the security and convenience of cloud-based iOS device backup.  After that though, competitors start to look a little nicer in the price & capacity range.  Both Amazon and Microsoft have decently-priced tiers that provide unlimited space.

So with the constraints of iCloud storage, they introduced their new Photos app.  (It’s not bad, but it’s no Aperture).  I tried importing and uploading my photos, but as you guessed it, I ran out of space.  I have about 10,000 photos, which is a lot but not a lot a lot, you know…  Now, iCloud Photos is by no means required to use, but having lost photos a few times to hard drive deaths, I decided I need some sort of cloud backup.  It would also be nice to save on the limited SSD space of our MacBook Air by keeping those photos primarily in the cloud.  But iCloud wasn’t it.

Enter Flickr

No not him, another one. Certainly © Disney.

I signed up for Flickr quite a while ago, although at the time it wasn’t really anything that compelling.  Then Marissa Meyer came along and kicked Flickr into the spotlight by giving 1TB away for free.  I instantly became a fan, yet I didn’t use it as much as I’d like.  There was a hurdle.  I could auto-upload from the iOS app, and that was wonderfully seamless (I was pleasantly surprised that many of my photos were up there already).  The hurdle, however, is the desktop.  When you import photos using a card or plugging your camera in… or you do some editing, then what?  I have a couple of plugins that allow you to upload to Flickr, and they work pretty good, but it’s a single-photo manual process… subject to my easily distracted mind (ooo something shiny!)

A few days ago, though, they came out with v4.0 of the iOS app.  I heard some raving about it, but the previous iOS app was already very very good.  Then I looked into the web – and more importantly – the Uploadr.  This is an agent that sits on your computer and does what it says.  It monitors your photo libraries and selected folders and uploads them to your Flickr account.  It deduplicates too, so you won’t have multiple unnecessary copies.

The web now has a camera roll that lets you do bulk operations like batch delete  (oops, I uploaded a screenshot folder) or album management.  It’s a little clumsy at the moment, since it falls back to the old interface in a few places, but already it’s very welcome.

Is it a complete “just works” solution like iCloud?  Almost, but not yet.

The one thing that is missing is the elimination or reduction of local libraries.  It would be nice to have an app that could access your hosted library, and optionally cache locally some full-resolution images, so you get the benefits of both cloud and local photo libraries.

Kinda like iCloud Photos.  Only affordable.

Linux Networking Server

Clone a Clone

So I had yet another WD MyBook die on me a couple of days ago.  And I still went out and bought another one (what was Einstein’s definition of insanity, again?)… This one was only two years old but these things are still quite cheap and very convenient to get.  Maybe someday when I have more money I will get a proper NAS enclosure.  For now, my pattern is to buy a new one every year.  They’ve almost doubled in size every time, so I can just clone everything to the new one and go from there.

Since I had such great experience with my WD MyBook Live, I decided to get the next version, a MyCloud.  This is slightly improved, similar in appearance, also with a GB network port but this one also has a USB 3.0 host on it, so I can buy another, regular drive next and slave it off of this one.

My previous scheme didn’t seem to work well with this one, though.  I was unable to make an unattended rsync to push from that drive to this one because these drives are set up to use root as the ssh user, and it’s not set up to use PermitRootLogin without-password .  It always seems to prompt for the password, which won’t work when AFK.

Until I figure that one out, I looked for another solution.  The coolest discovery was that these drives are actually running Debian.  After some research I found out that lftp will mirror a remote directory over FTP.  Of course, lftp is not installed on these drives, however, after running…

apt-get update
apt-get install lftp

I had them installed (on both the 2012 and 2015 vintage drives).

Next was the task of setting them up!  I found a good post on StackExchange (well, ServerFault) here that helped a lot.  I ended up using this:

EXCLUDE="--exclude-glob .Trash*/ \
 --exclude-glob .TemporaryItems/ \
 --exclude-glob .AppleDouble \"
lftp -c "set ftp:list-options -a;
open '$FTPURL';
lcd $LCD;
cd $RCD;
mirror --no-umask --only-newer --use-cache --parallel=3 \

With that I had some options I could use by uncommenting DELETE or TESTMODE, for example.  One additional gotcha is that it doesn’t retain the source’s ownership, but since this is such a basic setup, I just chown all the files in the LCD variable to my username.

The password for that user is in cleartext, in the file.  If you are not comfortable with that, do not use this.

It still doesn’t seem to be running in cron yet, but I need time to experiment some more.  I still much prefer the SSH method but I want to get it working reliably and repeatably.  I need to reimplement much of this each time the firmware gets updated, and copying a few files is much better than having to edit sshd config files each time.