Managing Encrypted files on Amazon Cloud Drive

I have implemented a file system on Amazon Cloud Drive for a lot of media with the great acd_cli.  To protect my privacy, I have run this through an encryption layer encfs.  My writeup will follow.

A problem I was trying to solve in my mind though, is how to manage – rename and delete files once they’re all scrambled up and I can’t discover even the path and filenames.

Ultimately this would be seamless.  Delete a local file stub and it traces back to the encrypted remote file, but it doesn’t quite work that way.  I discovered how to do this on my Linux host.

Once I realized that the filesystem for encfs has the same inode numbers for the encrypted and decrypted files, I had a clue.  First, let’s find out what that file number is:

149 is the part we want.  inode numbers are unique per partition/filesystem, and seems to persist between the encfs pairs.  Now, to find a file in the encrypted path system with inode 149… find to the rescue!

I won’t even try to copy/obfuscate the number above.  Try it if you want to see it.  It would be almost impossible to track that file without the number.  Size and date are much harder to nail down the exact file.

So, to stitch these two together first you want the inode number only:

Now this is something we can use in a delicious Linux command chain.

This is easy enough to make into a little bash script, and allow passing arguments and quoting to protect against embedded spaces, as well as including the explicit Amazon Cloud Drive working area:

Works great for specific files, not so much for directories.  You would have to change the ls command to use a -ldi parameter just for those cases.

Now that we have the filename, we can manually delete that filename on Amazon, either through the web interface or using acd_cli’s command line trash argument.

Clone a Clone

So I had yet another WD MyBook die on me a couple of days ago.  And I still went out and bought another one (what was Einstein’s definition of insanity, again?)… This one was only two years old but these things are still quite cheap and very convenient to get.  Maybe someday when I have more money I will get a proper NAS enclosure.  For now, my pattern is to buy a new one every year.  They’ve almost doubled in size every time, so I can just clone everything to the new one and go from there.

Since I had such great experience with my WD MyBook Live, I decided to get the next version, a MyCloud.  This is slightly improved, similar in appearance, also with a GB network port but this one also has a USB 3.0 host on it, so I can buy another, regular drive next and slave it off of this one.

My previous scheme didn’t seem to work well with this one, though.  I was unable to make an unattended rsync to push from that drive to this one because these drives are set up to use root as the ssh user, and it’s not set up to use PermitRootLogin without-password .  It always seems to prompt for the password, which won’t work when AFK.

Until I figure that one out, I looked for another solution.  The coolest discovery was that these drives are actually running Debian.  After some research I found out that lftp will mirror a remote directory over FTP.  Of course, lftp is not installed on these drives, however, after running…

I had them installed (on both the 2012 and 2015 vintage drives).

Next was the task of setting them up!  I found a good post on StackExchange (well, ServerFault) here that helped a lot.  I ended up using this:

With that I had some options I could use by uncommenting DELETE or TESTMODE, for example.  One additional gotcha is that it doesn’t retain the source’s ownership, but since this is such a basic setup, I just chown all the files in the LCD variable to my username.

The password for that user is in cleartext, in the file.  If you are not comfortable with that, do not use this.

It still doesn’t seem to be running in cron yet, but I need time to experiment some more.  I still much prefer the SSH method but I want to get it working reliably and repeatably.  I need to reimplement much of this each time the firmware gets updated, and copying a few files is much better than having to edit sshd config files each time.

More hack attempts

After my last experience, I checked my logs and noticed quite a load of failed attempts on my mail server.  It looks like a brute force script kiddie attack, which I’m pretty sure will fail on my machine.

Still, I want to kick out these morons.  So after some research, I found fail2ban.  The installation was simple enough, and with a little bit of configuration (in jail.local, not jail.conf!) I had it up and running… but the attacks continued.

I wrote a simple perl one-liner to parse out all of the failed login attempts, run them through sort and uniq to get the repeat offenders (twice is enough, kids) and append that to the hosts.deny.  That worked, but not ideal.  I’d rather have iptables-level blocking (using DROP instead of REJECT to waste as much of their time as possible).  But fail2ban wasn’t catching them for some reason.

I set up a secondary rule and it still failed – until I discovered fail2ban-regex! With that command you can test your rules at any time instead of waiting for the next attempt to come in.  Great!  It turns out the regex wasn’t quite right for the messages I was getting.  I simplified the regex until it caught the failures.  But it still wasn’t working live.  Grr.

fail2ban works on log files.  It scans for repeated attempts to determine if there’s an attack going on.  This would work great unless the logging daemon compresses the messages with something like “last message repeated x times”.  And this happens a lot, especially when under attack and you actually need it!  You can not turn this feature off with sysklogd.  The last key was to replace sysklogd with syslog-ng and POW, the banstick came out to play.

Debugging wasn’t very easy, because the failures are silent.  Until I found fail2ban-regex I had about 4-8 hours between tweaks to the regex to see if it worked.

At least now I have a self-setting ban trap that uses iptables-level blocking.

If you’re reading this and you’re learning to be a script kiddie,  you are learning to be a loser.  You are creating nothing of value.  You could vanish from the Internet and not only would it become a better place, but the situation would improve.  Is that really what you want?  Instead, why not keep on learning about security but do this the right way, on your own machine or a VM and learn to strengthen the Internet, not ruin it.  You might actually be actually appreciated and valued by others on the net.

Hack Attack

Someone mentioned they got a bounce from my domain’s email. I went to take a look at the error and discovered a couple of hosts trying to brute force login to my SMTP server. Some quick config changes to create a blacklist, and a fail2ban install and it has stopped now.

Lesson 1: check your logs often
Lesson 2: use SASL
Lesson 3: use complex and random passwords
Lesson 4: install and configure fail2ban or blacklist the bozos with iptables or hosts.deny or something.

I got most of these right the first try, especially the middle two.

Eternal vigilance, they say…

Great VPS Experience

I have been using shared hosting from 1 & 1 for a few years now. No big complaints (except their domain management page is terrible) but I needed a bit more flexibility. I suppose shared hosting with its inherent limitations helps you develop a certain way, but I wanted a bit more.

For one, I wanted a much more flexible IMAP server setup. I was getting something like 100 mailboxes at 2GB each, but what if I wanted to totally move off of gmail and use 5GB? Setting up one or two archive accounts seemed… Messy.

So I did some research and found lowendbox.com. It is a bit intimidating at first because you don’t know what to look up, or what questions to ask, like “what is OpenVZ, and does it matter for what I want to do?” Or “what happens when I lock myself out?” (Notice, that’s when, not if). “What about if I want to wipe it out and start fresh?”. “Do I need one of these control panel products?”. I have those answers now, if you want. 😀

Well, I didn’t have any very clear answers to any of these but I decided to try it out, with a great special from LEB for $56/year for a 2GB machine from ServerMania.

This. Stuff. Is. Cool.

If you have never done any admin work on Linux, set up your own server or anything, stay away. You will get little value out of it and very likely get hacked. But if you are confident with your admin and security chops, this is a crazy bargain. I chose Debian 6 because Debian. I have a long history with that.

I spent a good day figuring out, testing and setting up the firewall, and learned an amazing amount. I tightened up SSH, installed some IDS stuff and I was happy, even though the thing was hardly useful yet!

I used git during the setup so I would have some ability to track and undo dumb changes. I should have started this earlier, I know it!

First big task was IMAP and SMTP. I got that rolling and made sure that spammers couldn’t hitch a free ride. I got a free SSL certificate from StartSSL and it does the job. Test like crazy.

Next I got Apache rolling and a database server, and made sure those aren’t vulnerable either.

Finally, once I was happy, I pointed my domain to freedns.ws and configured its records. Let me stop for a second and state that of all the stuff I have done so far, FreeDNS.ws amazes me the most. I can’t see why anyone would pay for DNS server services when this thing is around (ok I notice certain record types are missing. I didn’t really need them though).

While waiting for the DNS switchover to happen (always an interminable wait) I discovered mydnscheck.com. A great tool to see how things are progressing and what you have set up or have forgotten to set up!

So far that’s it. I haven’t cancelled my shared hosting yet but I’m moving everything off it. I think I’ve outgrown it… Or more likely it just doesn’t suit me, as I never really used it to it’s potential.

Now I might even run a personal Minecraft server on my VPS, not sure yet. But hey, I can.

It was a lot of work, but a lot of fun. And I am happy in knowing there’s a little piece of the Net that I put together all by myself.

REST Authentication

I did some research the other day to secure my REST API using The Slim framework.

I found a tidy little way to force HTTP authentication (basic, in this case) using this article as well as the PHP manual.

I get the client to provide the user name and password, then I can look up the (hashed) password in the database. It simply causes the call to authenticate each login with their corresponding password.

In combination with forcing the page through https (a .htaccess task) and this, I can protect API access pretty simply.

I used this in VB6 (and the great Chilkat components) to authenticate a PUT command. I can now more safely allow database inserts from over the Net. It was as simple as:

Next task is to find a more secure way to access the (read-only) API from JavaScript, without just embedding a password in the source code.  This looks like a great start.

MVC Frameworks (PHP)

I’m on a learning quest again.  This time to get a solid PHP framework under my belt.  I am installing the framework and running the quick start tutorials for each of them, if I get that far.  This is what I have right now (keep in mind my investigation is in progress):

CakePHP

Pros:

  • Very easy to get a simple app running.
  • Scaffolding is great
  • Documentation is extremely thorough.

Cons:

  • Seems like database names have to follow some kind of convention. (i.e. Inventory would not work)
  • Getting REST JSON interface going seems like a lot of work.  I haven’t found a page that explains how to do it easily.

Zend (in progress)

Pros:

  • Insanely popular, the engine is already on 1&1 (I think)
  • Really good object-oriented approach, interesting idea with form & layout objects
  • zf code generator is a nice touch.
  • Looks like PHPStorm has the command-line stuff built in – also for Symfony

Cons:

  • Ridiculously hard to get running in a shared host. (Needs a vhost to work, apparently)
  • Lots of code.  Not a big deal, really, if you know what you’re doing.

CodeIgniter

Pros:

  • Scaffolding is neat, but REMOVED in 2.0
  • nice OO layout
  • Fairly compact
  • Looks like view names and table names are not hardcoded! Yay!

Cons:

  • PHPStorm doesn’t seem to like completing 3-level objects (lots of @property tags needed)
  • v2.0 scaffolding is gone
Right now, I’m liking CodeIgniter the best.  I am going to give Zend a lot more attention though, I think there will be a lot of advantages if I can figure it out.

Another Javascript book

I decided I had better complete my Javascript trilogy with the first book.  (The other two are Javascript: the Good Parts and Javascript Patterns) This is a major book for a major language.  I think I agree with this guy to a point, that Javascript is possibly the most important language right now.  It’s weird to contemplate but the language has taken kind of a stealth advance in recent years.

Of course it could be toppled very easily.  Visual Basic was probably the most important language at one point.

So far the book is extremely thorough, and touches on some new ECMAScript5 stuff that wasn’t addressed in “the Good Parts” (i.e. Object.Create stuff).  This is not really a reading book but I’m reading it anyway.

WordPress configuration

I need to figure out the following:

Is there a way to blend pages and posts together nicely on the site, so that I don’t have to configure only posts to see them on the main page?  I would like to make a menu that will show me maybe the latest five or so, but allow me to dig down further.

I know I can do it with PHP, but I want to know if there’s a configuration way to do it.

I found something.  Create a blank page and in the “Static page” section of Settings – Reading, choose the blank posts page to view the posts.

Snack: Dogfood

Well hello.

I am finally setting up this blog, not because I really need to share my thoughts with anyone, but because I need more practical WordPress experience.  Unfortunately that means I will have to include content as well as code.  Hope it is a) entertaining b) informative or c) nutritional for all of you.