Is Time Estimation in Software Engineering a System 1 or 2 Task?

Software engineers are notoriously bad at time estimation. When they receive a new bug report or product feature to work on, engineers are often asked by their project manager to guess how long it will take. It’s a very reasonable request. For example, if your website goes down, public relations needs to know how long it will take to put together a response. If you’re working on the next version of your app, product management needs to know what features are possible before the ship date. Continue reading “Is Time Estimation in Software Engineering a System 1 or 2 Task?”

Discovering My Web Design Mistakes

Although it is already February, I have just started my 2017 goal of learning web design. Since I prefer to learn through doing, I decided to start with my primary side project, Spawning Tool, to develop my skills. On Spawning Tool, StarCraft players can browse replays and guides submitted by other players to find strategies to try out in their own games. I have had a lingering concern about the home page content but couldn’t figure out what needed to change. Since that is the landing page for most visitors, it was a good place to start my design journey.

I asked my coworker Alex, a designer, how to learn web design, and he recommended that I look at other websites and pick out what design elements I like and don’t like. Following his advice, I picked out three sites similar to Spawning Tool in some way and compared their home pages.

Continue reading “Discovering My Web Design Mistakes”

A Beginner’s Guide to vim

(This was originally posted on GitHub. For the unfamiliar, vim is a text editor often used for programming. It’s similar to Notepad or TextEdit but much more customizable and powerful. It is also one of two sides in the great Editor war.)

Through your coding career, you have likely had brief encounters with vi while looking over your coworker’s shoulder or tweaking a server setup script. It looks arcane, and it’s hard to tell if it’s just an old habit of some coders or if it will actually make you more productive.

This guide is light on actual content and is intended to provide structure for how to learn vim. In truth, the only way to learn vim is to use it. The learning curve is steep but well worth it.

My perspective

vim is versatile, and this guide is written from the perspective an engineer who (mostly):

  • uses a Mac
  • ssh into Ubuntu virtual private servers (e.g. AWS EC2 instances)
  • writes Python/Django and JavaScript/React
  • builds a web application

Hopefully this guide is applicable to others as well.

1. Movement and editing

The basics of any text editor is being able to edit text, and vim is trickier than typing and deleting. There are plenty of cheatsheets and laundry lists of commands, so we won’t add another here. The best way to learn it is to usevimtutor, a tutorial built into vim that you can launch from your console. Go through it first to learn the basics, then do it again in a week after using vim for real.

2. Split Screen

We maintain a 100 character line length in our codebase, which is less than half of the width of a 1920px screen. Typically, I keep 2 different windows open side-by-side. You can split your screen with :vsplit and switch back and forth with ctrl-w ctrl-w.

Search online for more split commands (there are a lot).

3. Autocomplete

ctrl-p is your friend. When you’re typing, you can autocomplete the current word, where the options are drawn from words found in all open buffers.

4. Search and Search/Replace

vimtutor should have gotten the basics of find/replace, but here are some particular helpful techniques I have found

# and * search for the word currently under your cursor.

5. Editing your vim configuration

vim is extremely deep because it is extremely customizable. This includes simple settings such as tab widths or case sensitivity in search, but it also includes syntax highlighting by file type and rich plugins.

My dotfiles are available on github to see my settings in my .vimrc. There are 2 semi-standard settings that have been tremendous for me personally:

  • imap jj <Esc> – Type ‘jj’ to exit insert mode. This saves the stretch for the escape key or for ctrl-{
  • nnoremap ; :, nnoremap ; : – You type colon far more often than semi-colon, so this swaps those

Currently, I would recommend Vundle as your plugin manager.

7. GNU screen

Technically this is outside of vim, but I find screen to be indispensible in my environment. screen provides virtual shells so you can rapidly switch between vim and the shell. My typical setup has 3 screens:

  1. vim – I only have 1 instance of vim running at any time to avoid conflicts. I use multiple buffers and vim-session to maintain that
  2. shell – everything else I use outside of vim, including ack, git, scripts, file system management, etc.
  3. server – for web dev, you usually have to run a Django/Rails/Node development server, so I just keep that running in a separate shell

Note that you can also create a .screenrc to customize your screens as well.

8. Macros

Macros allow you to record a series of commands and repeat them again. Very useful for repetitive tasks

9. Marks

Marks record your cursor position, which you can use as a “bookmark” or in combination with other commands. I usually use uppercase marks because I’m bouncing between so many files

10. Registers

Registers are basically a multi-clipboard. I like using registers when I am replacing repetitive code since the unnamed register keeps getting overwritten when I delete lines of code.

11. References

  • Jim Dennis on grokking vi – a long but also deep StackOverflow answer on how vim works
  • Vim Tips wiki – it’s not very well-organized, but you will stumble across this site often while googling for vim tips
  • Vim Awesome – index of vim plugins
  • List of Django dev tools – for life outside of vim, here’s a full stack of tools you can use for Django development

12. So much more!

Although this guide is a quick read, learning to use vim can take a long time. While you are coding, be aware of the tasks at hand and keep that meta-awareness about what vim tricks you might use to be more efficient. Over time, more of it will become automatic, and everything will feel quick and intuitive.

Beyond this guide is a world of customizations and additional shortcuts to continue to learn.

Treasoning about Code

A few months ago, I switched from emacs to vi.

This is a really big deal, so please take a moment to process this fact.

If you don’t have a strong opinion on this matter, let me explain what the big deal is in the editor war.

People have lots of tiffs in technology choices. The most common choice is Windows versus Macs, with the latter having a fanatical user base willing to argue strongly in the superiority of their operating system. In the coding community, we argue about the best text editor to use. Programmers spend more time in their text editor than anywhere else because that’s where code is written. Although we could use simple editors like Notepad or TextEdit, we prefer to have more powerful tools. Just as Microsoft Word is a better word processor because of spellcheck and formatting tools, editors like emacs and vi are better text editors because of autocomplete and syntax highlighting.

There are, of course, many different choices of editors, with much bigger differences you would see between Word, Google Docs, Pages, or other word processors. Some, such as Eclipse and Visual Studio, have a full graphical user interface (GUI) with multiple windows, menus, and buttons visible to use. These editors visually are similar to modern word processors. Although these environments are powerful, many developers such as myself, prefer to work from command line editors, which still have the blocky, text-only appearance of things like MS-DOS. They definitely have a steeper learning curve, but the 80s-like interface still has its advantages. The primary advantage is more extensive reliance and support on keyboard shortcuts (think “Control-C” instead of clicking the “Edit” menu and “Copy” option for copying). Programmers can keep their hands on the keyboard the entire time instead of switching over to the mouse.

Within this world of command line editors, there are 2 primary flavors from which the rest are derived: emacs and vi. Both offer a lot of power from the keyboard, but they were built very differently, both conceptually and technically. The biggest difference for regular usage is that vi is modal, but emacs is not.

vi is modal because keys will perform different actions depending on which mode the editor is currently in. The most common mode is “insert-mode”, where typing letters will insert characters, like any regular text editor. If you exit this mode (by hitting “escape”), you return to “command-mode”, where many other actions are available. For example, “Y – Y” will copy the current line, and “P” will paste it.

emacs is not modal because there aren’t multiple modes. To add additional actions on top of letters, the user holds down modifier keys, just like in word processors. For example, once you have selected some code, “Alt – W” will copy that code to the clipboard.

The differences go far beyond that, but in short, programmers have irrationality strong opinions about which editor is better, which is usually based on first exposure, not reason. For example, Stanford mostly generates emacs users since emacs is introduced in the 3rd programming class, whereas vi is never formally taught. That’s where I started, and only the next 5 years, I learned more shortcuts and became a faster coder.

I eventually hit a roadblock with emacs, though, and it was physical, not mental. I code all day for my job, and I spend a lot of time on my computers in the evenings. Between all of that, my fingers really started to hurt. As I mentioned, emacs uses modifier keys for various actions, which is a lot of holding down “control” and “alt”. I tried to reduce the awkward hand positions as much as possible: I swapped the “control” and “caps lock” keys on my left hand so I had easier access to “control”. I learned to use both “alt” keys for different combinations to split the workload. Despite these adjustments, however, I still needed to both stretch my pinkies and tuck my thumbs to hit keys, and that wasn’t getting better.

So I overcame the ridiculousness of a modal editor (where keys do different things without telling you when such is the case) and tried out vi. It’s been a few months now, and it’s turned out well. I don’t think I’ll ever be as fast with vi as I was with emacs, but more importantly, my hands don’t hurt. I still use emacs occasionally (for example, it integrates with the python debugger really well), but I’m fully capable of switching contexts and my mental set of shortcuts to use both.

I’m glad I can use both. It’s like being ambidextrous, or being a pitcher and catcher. But more importantly, I feel really awesome. Programmers who use command line editors generally think they’re superior to programmers who use editors with GUIs, and being able to use both of the major command line editors is even a step above that. So switching wasn’t just to help out my fingers: it’s been good for my ego, too

Installing Django on Mountain Lion (+ MySQL and PIL)

UPDATE 1/14/13: A few small corrections. See Jonathan’s comment below if you’re having issues installing mysqldb

A few years ago, I posted instructions on installing Django on Snow Leopard. Since then, things happen gotten a lot easier. At that time, there was a weird shift going on with 64-bit computing that made it difficult to know what versions of various software to install. Well, it’s a few years past that, and package installers have become much easier to use as well. So, if you’re looking to do a fresh Django install on your computer, take a look:

1. Install MySQL

Download MySQL from http://dev.mysql.com/downloads/mysql/. Specifically, you want the “Mac OS X ver. 10.6 (x86, 64-bit), DMG Archive” version. Install mysql*.pkg as given. If you would like, you can also install the startup item (if you want MySQL to start automatically) and the preference pane so that MySQL can be started from System Preferences.

For some convenience, you can also add mysql to your path to invoke it in Terminal directly. To do this, open ~/.bash_profile with your favorite editor, and add

export PATH="/usr/local/mysql/bin:$PATH"

to it. You should also set your root password for MySQL using

/usr/local/mysql/bin/mysqladmin -u root password YOUR_PASSWORD

References:

2. Install libjpeg

Feel free to skip this if you don’t need the Python Imaging Library, but if you do, homebrew is the easiest way to get it and many other packages that you deserve as a UNIX user. Long ago, I used fink, and it was an awful experience. I hope they’ve improved it since, but homebrew was so painless.

ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)"
brew install libjpeg

References:

3. Install virtualenv (recommended)

I hadn’t used virtualenv before doing this setup, but it’s great. In short, it creates a virtual environment to setup python packages. Even though it’s easy to install packages otherwise, it does become a mess if different applications have different dependencies or if you need to swap python versions. This does away with all of that.

sudo easy_install virtualenv
sudo easy_install virtualenvwrapper

Then start from line 2 of https://gist.github.com/1852087 to get your virtualenv setup.

References

4. Install the python packages

If you didn’t install virtualenv, you’ll need to install pip (sudo easy_install pip) since pip otherwise is packaged with virtualenv. You will also need to sudo the next steps.

I recommend that you use a requirements file to setup your packages. It probably isn’t a big deal if you’re working on your own project, but it’s immensely handy as soon as someone else needs to setup the same environment as you. To do that, create a requirements.txt with contents similar to

django>=1.4
MySQL-python
pillow>=1.7.8
South>=0.7.6

pillow is optional depending on if you want PIL, and so is South. If you have your own favorite way of doing schema migrations, feel free to substitute. If not (or if you don’t know what South is for*), I recommend South. After you have that setup, just run

pip install -r /PATH/TO/requirements.txt
sudo ln -s /usr/local/mysql/lib/libmysqlclient.18.dylib /usr/lib/libmysqlclient.18.dylib

I don’t exactly know what the circumstances are for requiring the second line, but I needed it, so I’m going to assume you will too.

References:

And that’s it! Now, you can fire off your Django project from within your virtual environment for local development. If you mine through some of the other links above, you will find a lot of other really good instructions for setting things up. Presumably, you can also google things as necessary.

Let me know in the comments if you run into any problems, if I missed anything, or if you have any proposed enhancements to this process. Also, let me know if I can do any other advocacy on the part of Django. I have had largely good experiences with it for building websites.

 

* one thing that Django doesn’t address well is database migrations. When you setup your models initially, Django will create db tables for the fields you specify. As fields change, however, Django doesn’t deal with changing the db tables. A tool like South will analyze your models and create migrations for you to run and modify the db schema

The Board Game Chooser: find a board game for any occasion with a few simple questions

My friends and I have been playing a lot of games together recently. It’s quite remarkable how much time one has when no longer in school, and given how much time we spend together, it always helps to find fun, accessible activities for us to do together. Enter board games.

It turns out there’s a huge world of board games. BoardGameGeek is a great database of the ones that exist, but with so many and a decent investment to get any of them,  it’s hard to know which one is right. Right now, most of them aren’t right since my housing is in transition right now, but if I wasn’t, it would be really handy to find a way to narrow down to my own preferences in games and the situations I would be playing it in. For that, I built the Board Game Chooser, a simple website that walks you through to a good fit.

All of the credit for the data goes to the Silver Oak Casino, who made a huge flowchart for picking a board game. My contribution here was just processing the data and putting it online for greater accessibility. I hope it’s helpful to you.

For a project that took only a few hours, I’m happy with how it works. All of the data is in JavaScript, so notice that once you load the first page, you never need to load another page again. Not only does this reduce the amount of traffic my server needs to deal with, it results in a much faster experience for you as well.

Given that the path does feel like a series of pages, however, I am also using browser history in HTML5 to simulate the same effect. If you’re in a modern browser, you’ll notice the URL change as you click through choices, and if you navigate directly to any of those pages, you’ll return to the same state. It even supports the browser “back” and “forward” buttons, which have become critical in surfing the web but can often be a web developer’s nightmare.

Anyways, enough about tech. If you’re still interested, you can see all of the code at https://github.com/StoicLoofah/boardgame-chooser. Otherwise, go ahead and try it out, and I hope you find something that interests you.

And as a last second pitch, if you do want to play a game, please consider going out to a local game store or bookstore instead of just buying it online. The game store I went to growing up was a great place, not only as a retail space for fun things but also as a community for people looking to play games of any sort. Internet retailers may be $10 less than MSRP, but they don’t have tables in the back to meet new people to play with.

Introducing chronoline.js, a JavaScript library for timelines

Sadly, a lot of my work at Zanbato is behind closed doors, but recently, I have been working on a widget that is definitely not proprietary and is available for all of you to use, extend, or even just look at: chronoline.js.

chronoline.js is a library for making a chronology timeline out of events on a horizontal timescale. From a list of dates and events, it can generate a graphical representation of schedules, historical events, deadlines, and more.

This is just the short pitch, but hop on over to http://stoicloofah.github.com/chronoline.js/ where you can see a few examples. They don’t quite demonstrate what I think is quite a bit of built-in flexibility, but definitely let me know what you think!

Installing Django on Snow Leopard (and MySQL and PIL)

UPDATE 11/3/12: If you’re now on Mountain Lion, life has gotten much easier. Read my new post on it

UPDATE 8/4/11: The instructions here are a little dated and difficult, probably from my own ignorance. With Lion out, too, things have changed enough that you should probably use the instructions here instead.

I recently needed to port a Django app off of our Linux server onto my Macbook Pro, which was slightly more involved than I thought it would be. I don’t remember having nearly so much difficulty with it when I initially set it up on my old Leopard partition, but times change. In any case, I figured it would be a common enough task that I would offer instructions on how to make it work. If you’re looking to just install Django fresh, you’ll notice there are a few steps specific to importing parts from another Django app, but it’ll mostly apply. So let’s go.

Here’s exactly what we’ll be putting together:

  • MySQL
  • Django
  • MySQLdb – a python package to communicate with MySQL
  • libjpeg – necessary for PIL
  • Python Imaging Library – a pretty useful library for drawing and more

If you’re more slick from Terminal than I am, there are a lot of steps here that you can do out of that. If that’s you, then you can translate those operations yourself. If you’re like me and still like GUIs for some tasks, maybe this will work better for you. I’ll also leave a trail of links to the references I used to figure it out myself so you can read the real reference

1) Installing and setting up MySQL

Download MySQL. Particularly, I ended up getting the 32-bit version for reasons that are not clear to me, though the 64-bit version might have been the correct call. I at least know it works with the 32-bit version. There’s a .dmg version that apparently fast-tracks the installation, but I ended up not using that either because it didn’t download properly. So here’s what you do to do it manually from the tarball.

and unzip it by just double-clicking on it in Finder. From Terminal, run the following

cd /usr/local
sudo mv (copy path) ./
sudo ln -s full-path-to-mysql-VERSION-OS mysql
cd mysql

Here’s a nice trick I learned recently. You’ll notice the code above has (copy path) in it for the path to the mysql folder you just unzipped. Instead of typing it in, you can click and drag the folder from Finder to Terminal, and it’ll fill in the whole path for you.

So if you want to do things properly, you’ll create a separate mysql user, apparently, and let everything run through that. In my case, I’m not planning on ever using MySQL locally for actual production, so I lazily just have everything running from my user account. To complete setup

scripts/mysql_install_db
bin/mysqld_safe &
mysqladmin -u root password NEWPASSWORD

Where you can fill in the root password. Next, for convenience, let’s add the mysql commands to our path so we don’t have to type them every time. Open up your bash profile and add this line (if it doesn’t already exist, just create it at ~/.bash_profile)

export PATH=./:/usr/local/mysql/bin:$PATH

Of course, if you already have a path there, you only need to append the bit about mysql from it. For convenience in the rest of this setup, copy the same line into normal Terminal and run it to update your PATH variable in session.

Right now, the mysql database is running because of “bin/mysqld_safe &” above. In general, from now on, the command to start it is

mysqld_safe &

and the command to shut it down is

mysqladmin -u root -p shutdown

where you’ll need to type in the root password you defined above. Go ahead and start the MySQL database again. The next little bit is for porting the contents of a web server database to your local machine.

mysql -u root -p
create database DBNAME;
exit;
ssh username@server.com
mysqldump -u root -p DBNAME > ~/DBNAME.out
exit
scp username@server.com:~/DBNAME.out LOCALPATH
mysql -u root -p DBNAME < LOCALPATH/db-name.out

Reference:

  • http://dev.mysql.com/doc/refman/5.1/en/mac-os-x-installation.html
  • http://dev.mysql.com/doc/refman/5.1/en/installing-binary.html
  • http://www.cyberciti.biz/tips/howto-copy-mysql-database-remote-server.html

2) Installing Django

This part is thankfully very easy. Download Django from here and unzip it. I myself ended up using the 1.0 to avoid compatibility issues with our web server, but I’m guessing the latest version is the greatest. cd to the folder you just extracted and execute

sudo python setup.py install

Boom, done. Thanks Django!

3) MySQLdb

Okay, so this is going to let us tie the 2 pieces we’ve put down together. Download MySQLdb from here, unzip, and navigate to that folder. In Terminal, execute the following:

 

ARCHFLAGS='-arch x86_64' python setup.py build
ARCHFLAGS='-arch x86_64' python setup.py install
defaults write com.apple.versioner.python Prefer-32-Bit -bool yes

To check to see if it worked, execute this:

python
import MySQLdb
exit();

So if you didn’t get icky output, then that worked. If it didn’t, I’m not the person to ask.

If that worked for you and you’re not using PIL, then you’re done. Head on over to the Django tutorial if you want to try it. If you’re doing drawing stuff, then you’re not out of the woods yet.

Reference:

  • http://yousefourabi.com/blog/2008/06/django-on-leopard/
  • http://cd34.com/blog/programming/python/mysql-python-and-snow-leopard/
  • http://www.jaharmi.com/2009/08/29/python_32_bit_execution_on_snow_leopard

4) Installing libjpeg and Python Imaging Library

First, you’re going to need to have Xcode installed so you can compile lipjpeg. If that’s not already installed, you can find it on your Snow Leopard installation dvd under “Optional Installs” or from the Apple website.

We’ll start with libjpeg. Download it from here. You’ll probably want one that is of the form jpegsrc.v7.tar.gz, where the version might be different. Again, extract, navigate in, and execute this

export CC=/usr/bin/gcc-4.0
cp /usr/share/libtool/config/config.sub .
cp /usr/share/libtool/config/config.guess .
./configure --enable-shared --enable-static
make
sudo mkdir -p /usr/local/include
sudo mkdir -p /usr/local/lib
sudo mkdir -p /usr/local/man/man1
sudo make install
sudo ranlib /usr/local/lib/libjpeg.a

Don’t ask me what a lot of that means. I just copy what people tell me to do.

Finally, we can install the PIL. Download it from here, extract, and navigate it. The last Terminal commands are

python setup.py clean
python setup.py build
sudo python setup.py install

And that should be it.

Reference

  • http://stackoverflow.com/questions/1438270/installing-python-imaging-library-pil-on-snow-leopard-with-updated-python-2-6-2
  • http://jetfar.com/libjpeg-and-python-imaging-pil-on-snow-leopard/
  • http://snippets.dzone.com/posts/show/38
  • http://stackoverflow.com/questions/1398701/problems-with-snow-leopard-django-pil

Wrapping Up

So there you go. To get your apache server running to make your Django app go, go to System Preferences->Sharing->check “Web Sharing”. Your Django app can live where it is, but if you need to drop media files somewhere, the www root for Mac OS is /Library/WebServer/Documents/

Have fun!

Don’t rsync /var with –delete

About 10 minutes ago, I learned an important lesson: make sure you fully qualify the folder you want to sync. And don’t use the delete option if you’re not entirely sure how to use it. The bad news is that I accidentally deleted this website and panicked for a second.

But I like to look at it on the bright side. For about 10 minutes of downtime, the good news is:

  • There’s a reason why people don’t just use root all the time. I need to be better about this, because it paid off majorly this time to not have permissions
  • my laziness in working on my website template means that I didn’t accidentally erase a bunch of work
  • I now know how rsync works a bit better
  • It caused me to backup my blog database so that the next time there’s a disaster, I’ll only be partially screwed
  • I didn’t need to upgrade wordpress because a fresh install does that, too

Wow. That could’ve been a lot worse.

Sorting Algorithm

I don’t usually blog like this, but I had a semi-moment of inspiration, only being semi- because it might be complete crap.

Well, I was thinking about sorting when I thought about using math to quicken the process. This most obviously works with numbers, though I think it can be used for other stuff as well.

Basically,

1) Calculate the mean of the array
2) Find the standard deviation of each member
3) Using that number, put it into a general position in a new array, as its SD should give a general percent of where it goes
4) Deal with collisions by sliding left or right to a new position, or in some more ingenious way

After that, it can either go down as a divide-and-conquer, or just be a general sort for an insertion sort to follow or something.

So: incorrect, unusable, already thought of, or a combination of those?