weekends are for leisure

It Starts with Florida

Just read the Nov/Dev 2011 Audobon magazine article, Life Support, about the improving state of Everglades since the author’s previous visit 10 years ago. I won’t go into the details of the article. Nor will I pretend to have a solution for that region. However, the article conjured up some environmentally-related memories from growing up in Florida, so here goes:

As a child I was frequently puzzled by the fact that nobody knew exactly what to do with their old engine oil. Most people I knew dumped it in some far-away back corner of their property. They’d even joke about it probably not being the best thing to do, but they’d then comment on the low quantity and low frequency. And of course, I’ve done the same on many occasions. After all, I was in charge of mowing the lawn, which necessitate oil changes and removal of old gasoline from the previous season. I’d dump it reluctantly, because there didn’t seem to exist any alternative.

In 2001, when I bought my first vehicle I began to change my own oil. While purchasing oil and a filter I asked an employee at Discount Auto Parts about used oil. He told me there was a large blue container in the back that I could dump it in. Finally!

Truth be told, I have no clue what engine oil really does to the soil, or the organisms living in the soil, or the water table below. But that doesn’t matter. The possibility never sat well with me. I was just happy I didn’t have to play the cognitive dissonance game with myself. No more wanting to care about the environment while potentially damaging it and then deciding I’m not that big of a hippie after all. That sort of behavior (mental gymnastics) doesn’t help anybody. Moving on …

A few years after high school I saw a classmate who’d been living in a larger city after college. She had a job with some sort of environmental consultant. The news of that made me happy, but the way she described her job ruined my mood. I’m paraphrasing, but I recall her describing it as “making sure people get to use their land in the way they want”. I’m not sure whether my face showed my disappointment, but I felt horrible nonetheless.

There’s so much danger in letting everyone have free-reign over their land. A piece of property comes with plants and animals that depend on it, in numbers hard to comprehend. The world is complex, and it’s at our mercy. Not enough of us are considerate.

However, property-rights would be a lot easier to debate about if it weren’t for these problems:

• It’s hard to educate or inform others without being abrasive
• It’s hard to suppress the desire to stand your ground when another person suggests you might be in the wrong
• It’s hard to make a person care
• As humans we often think we’re the most important things on the earth, even though we’re not endangered in any way (but from ourselves)

We’d make much more progress if these four items weren’t such huge barriers.

Tag stripping not sufficient to prevent JavaScript injections

In the PHP world, solely relying upon PHP’s strip_tags() function to protect your web application from JavaScript injections is a bad idea. If you do, you may be vulnerable in even the most recent browsers (I tested in Chrome 17.0.963.83, Firefox 9.0.1 and Internet Explorer 9). There may be parallels in other languages too, so beware.

You’ll be vulnerable if the following are true:

• You’ve got a webapp that accepts user input
• You use strip_tags() or similar to sanitize fields
• You don’t explicitly remove less-than or greater-than characters from those fields (PHP’s strip_tags won’t remove a partial “<script” tag)
• Values from two or more of these fields are printed close to each other in the output HTML, with little or no markup between them

The fourth item is tricky … The markup between the field values must not contain any quotes (the quotes would prematurely close the script tag injection attempt). In other words, it’s possible for an opening SCRIPT tag to be constructed using values from two subsequent user-input fields.

Granted, the vulnerability only arises if markup is formatted in a very specific way, but it’s worth taking another look at your code. See this gist with example HTML for what it looks like to your browser.

I submitted a Chromium bug report, but it’s something they’re not interested in fixing.

Guess we’re on our own.

Not certain

Watched a 2005 Oregon PBS special about forest management last night. Link here. Saw some old guys with lots of land and trees managing it with intelligence. They don’t clear-cut. They have a clue about genetics so they save the best trees and sell the others. Made me think: Damn that looks like fun, rewarding work!

Especially the bits involving data collection. One guy kept detailed logs and found that his trees put on mass much faster later in life, but most loggers harvest before that point. I’d love to pay attention and keep those kinds of numbers. The data loving techie in me is thinking: GPS for each tree, monitoring CO2 and other levels around my property. Of course, I wouldn’t want the financial pressure of that being my only source of income. Luckily I’ve been a telecommuting programmer for years … been fortunate.

The downside is that I like living in the city. Love the density, the energy, the white noise, the grocery store 1.5 blocks away, the access to varied restaurants and lattes, the ability to walk for hours and see new parts of town. And of course, I don’t enjoy driving.

Something to think about.

A dispatch from the Moose Ear Diner

Completed Puzzle Agent recently. It was such a fun game and luckily the sequel went on sale a few days after I finished it. The puzzles were a balanced mix of 10-second teasers, those where your only hope is to jump in and work through possible solutions, and those where you really need to pay attention.

Aside from the puzzles, the other aspects of the game are no slouch either. The story draws you in and has quite a few surprises. The art work is simple, high quality, and infused with a playful sense of humor. Can’t wait to start the sequel.

The Thin Man

The Thin Man is a great movie. It’s filled with lovable characters, great pacing and a great plot. And of course I must mention the hilarious, witty protagonists. Luckily it’s only the first in a six-part film series. Looking forward to the rest!

Character encodings in practice

Building upon Joel’s post on Unicode, here are some real-world tips relating to character encodings.

Use them by name

Always explicitly specify which encoding you want (perhaps UTF-8). Don’t assume the language or library/tool you’re using will make the right decision for you. If you value your time, don’t ignore this recommendation, otherwise you’ll likely spend lots more time patching things up in the future.

A tale to drive this home

Until very recently, MySQL’s default charset was latin1. Our legacy code didn’t specify an encoding in PHP’s connection to MySQL, in our HTML, nor in our HTTP headers. And web browsers default to god knows what (“it depends”).

When the time came to export some table data to XML I ran into issues of invalid UTF-8 characters. I knew there were special characters in some text fields introduced by Microsoft Word copy-pasters, so I figured flipping the switch to UTF-8 would help. NOPE!

It appears that having failed to specify an encoding in the past caused special characters (fancy double-quotes, crosses, long dashes, etc) to be stored in MySQL in a fashion that doesn’t satisfy the UTF-8 format. Shit!

So I had to come up with a batch of find-and-replace regex patterns to turn those characters into proper UTF-8 or their HTML entities.

So please, do the following:

• Choose an encoding and stick to it
• Save your text files in that encoding
• Set the character encoding in your HTML
• Send the Content-type header with charset from your webserver
• Change the configuration defaults for MySQL or your DBMS
• Check and set the default for each of your databases (note I said database not DBMS)
• Check and specify the encoding for your tables and text fields
• Make sure you specify the encoding when you create or alter tables in the future

Don’t convert blindly

If you failed to do the above at one point in time, you may have content in your database that isn’t valid UTF-8 (assuming you want to convert to that to prevent further issues).

Users that copy and paste from Microsoft Word are likely the source of this problem, but don’t blame them or Word. Blame the programmer responsible!

Your task now is to convert those invalid UTF-8 byte strings into something more usable. You may be able to convert many to their HTML entities, but you probably want to convert alphabetical characters to their valid UTF-8 representation to ensure the text remains easily searchable.

Don’t convert blindlier

If you have to convert your table definitions and data to UTF-8 (from something like MySQL’s latin1), be careful. There’s a great post by the Wordpress crew of what it entails (multi-stage process, to intermediate binary column types first), but it didn’t cover this troublesome gotcha: If MySQL finds characters it can’t convert to UTF-8 it’ll truncate the rest of the field data

MySQL will gladly go through the motions with you as you convert from latin1 to UTF-8. It’ll issue warnings that something went wrong during the conversion. But it will still truncate your data at any character that’s not valid UTF-8. Instantly your article text will go from 10234 bytes to 145. There’s probably a valid technical reason why it doesn’t error out altogether, but all I know is I now have to re-populate tons of data from backups.

node.js FTP server

Got to use node.js for a work project recently. We needed an FTP server with special user authentication that would run custom code after a file was uploaded. There was one node.js FTP server implementation on github, so I forked it and started rounding out the basic functionality. My fork is here.

The first significant change I made was to encapsulate the data connection logic. File lists and file contents are transferred over the data connection (FTP commands and responses over the control connection). I quickly found that some clients are super eager to send you data and will do so once a passive data connection is made, even before the FTP server tells them it’s ok to do so. This was especially problematic for file uploads over passive data connections. Without a workaround for these aggressive clients, the flow looked something like this:

• Receive PASV command from client
• Start listening on a port and tell client which port to connect to
• Receive STOR command from client, stating it’s going to upload a file
• Attach data listener to data connection that saves incoming data to file
• On data connection end event, make sure file data has been written
• Close the file

Due to the asynchronous nature of node.js, data often arrived between steps 3 and 4, before a data event listener had been attached. Some of the file chunks were falling through the cracks and the saved file was incomplete.

Also regarding the above flow, ensuring that all data had been written to disk before we attempted to close file was … well … convoluted. Maybe a solution could be found using fs.createWriteStream, but I’m still happier with what I’ll outline next.

Once I found out that data was falling through the cracks I did some searching and found that setting up a buffering data handler was a common solution to a common problem. Good to know! With a persistent data buffer in place things became much easier, like so:

• Receive PASV command from client
• Start listening on a port and tell client which port to connect to
• Once we get hint of a connection (socket connect event), listen for data events and push each data chunk onto a stack
• Receive STOR command from client, stating it’s going to upload a file
• On data connection end, open file
• Loop over buffered data
• Save each buffered chunk before moving to next chunk
• Close the file

It’s less than ideal to buffer each uploaded file in memory first, but it works and is simple. Success!

In addition to the above, I’ve encapsulated things further in my forked repo. The FTP server object itself emits some additional events which you can listen for. My goal was to encapsulate the basic FTP functionality and provide a way to take special actions when:

• A user connects
• A user attempts to log in (listen for these events and handle authentication yourself)
• User uploads a file

Hopefully someone with a slightly different use-case will come along and take things the rest of the way.