weekends are for leisure

Problems Solved in 2012

Tags: #Programming  #Work  #Aws  #Php  #Redis  #Mysql  #Node.js 

Wherein I recap the things I’ve done and learned this year as a software developer (mainly at work). I’ve fixed many bugs and uttered many curses. Through it all I’ve constantly strived to make the code a better place.

Amazon Web Services

Node.js

Continued to make our video upload handler more robust. The aws2js library, which we use to transfer uploaded videos to Amazon S3, essentially does an emit bomb for multi-part transfers to S3. It splits the file in chunks (without enforcing an upper limit) and attempts to upload all the parts at the same time. This causes node to crash when processing large (>800MB) files, which we encounter quite frequently. To combat this I extracted the multi-part upload portion of aws2js and created a version that uploads file parts sequentially, with retries.

I also updated nodeftpd to use pipe() for file copying across streams, rather than copying buffers explicitly in the code. It was a simple change, but I enjoy cleanups.

Sphinx

We no longer have to daily re-build our rankings index thanks to Sphinx’s real-time indexes. Took a while to import, but the ability to insert/update/delete documents via SQL is super handy. Sphinx is a great tool.

JavaScript and UI

Experienced first-hand what it means to implement a separation concerns on the client-side. A few take-aways:

PHP

Redis

Memcache

MySQL

ffmpeg / libav

Sometimes videos will fail to convert for strange reasons. If so:

Sysadmin

NON WORK

Android

Other