blog.philcrissman.com - Home

Back To Wordpress

2008-04-03T18:16:00Z

I’m going back to Wordpress, for awhile, I think. I kept telling myself that the features I’d like to see in Mephisto, I would add. I think I could but I’m beginning to realize that I really don’t have the time, and I won’t have time for awhile.

So, back to Wordpress. Thanks for reading,

Social Networking and the handshake problem

2008-04-02T15:29:00Z

There was a lot of discussion awhile back about Facebook and its limit of 5,000 friends. The discussion, in case you (mercifully) missed it, was largely driven by social networking power users like Scoble.

I didn’t think too much about it at the time. Between now and then, I’ve thought a few times about social networks, and how they are implemented. Not the front end, but the back matter; the model. The database; users and connections.

A social network is fundamentally a graph. Most social networks are analogous to non-directed graphs – that is, there is no distinction which “direction” a connection goes. If you are friends with someone in Facebook or MySpace, they are also friends with you. Pownce and Twitter have taken a directed-graph approach, in which friendship (following) is not automatically mutual. I could follow you, but you might not follow me. And so on.

Like any data structure, there are multiple ways of implementing a graph. Fundamentally it consists of vertices (the users, or nodes) and edges (the connections between the vertices). If we need to represent a directed graph (like Twitter) we have potentially two edges between any two vertices. In a non-directed graph (Facebook, LinkedIn, etc) only one edge will suffice. As far as implementation goes, a common solution would be to have a table for vertices(users) and a table for edges(connections or friendships).

The Facebook limit of 5,000 friends poses an interesting question. What exactly is the overhead of 5,000 friends? If only a few people have 5,000 friends, it’s not too bad; 5,000 edges. But if you’re implementing this, you need to consider a worst-case scenario. What if everyone decides to have 5000 friends?

As a simple example, suppose 5,000 people in Facebook are all part of the same organization, and all decide to befriend each other. Let’s say that everyone in the Oracle network decides they need to connect to every other Oracle person on Facebook. For simplicity, let’s say this network has 5,000 members, so we don’t need to worry about breaking Facebook’s limit.

Assuming that the users and connection are stored in two tables as mentioned earlier, our “users” table would have 5,000 records.

How many records would the “connections” table have?

The answer is simpler than it might seen; it’s the same as the handshake problem. The handshake problem asks, if there are N people in the room, and everyone shakes hands with everyone else one time, how many handshakes have there been? The answer is the triangular number for N – 1.

Triangular numbers are like a factorial, but using addition instead of multiplication. So where N! would be 1 * 2 * 3 * 4 * ... * N, the triangular number for N would be 1 + 2 + 3 + 4 + ... + N. We’ll call it T_n. By this reasoning, T₁ is 0, T₂ is 1, T₃ is 3, T₄ is 6, and so on. For small graphs it’s very easy to confirm this by just drawing a few dots, drawing a line between every dot and counting the lines.

Is there a proof that for a graph of N vertices we’d need T_{n – 1} edges to connect all of them? I’m a little rusty with formal proofs, but this one is fairly simple. If you have N nodes, you could make sure they are all connected by starting with one, and then drawing a line from it to each other node; next you would move to the next node, and draw a line to every other node to which was not already connected, and so on, until you run out of nodes.

If you were to actually do this, your first node, N₁ would have N – 1 lines, as there would a line between it and every other node, excepting itself. When you move to N₂, it will only need N – 2 lines, as it is already connected to N₁... each consecutive node N_k would need one less edge to connect it to the remaining nodes, and you would wind up with edges numbering (N – 1) + (N – 2) + (N – 3) + ... + 1, or T_{n – 1}. Okay, as a formal proof it’s sloppy, but hopefully you can follow it.

Fortunately, there’s a simple formula to calculate T_n : n(n -1)/2.

So, if 5,000 people are all connected to each other, that requires 12,497,500 connections.

Okay… one step back here. Facebook has (last statistic I could find) about 67 million users. That means if each and every one of them were connected to their maximum amount of friends, it would require a database table of 167,466,500,000 records to store all the connections. Yes, that’s nearly 167.5 billion.

Not only that, but if the maximum friend count was increased by only one more friend each, to 5001, and every Facebook user took advantage of it, that would actually add approximately 133 million connections to the table. Just by allowing ONE more friend per person.

Now, I’m not a DBA, but those numbers are huge for database tables. I’m pretty certain that most commercial databases would need to split a table like this up into multiple tables, for performance reasons if nothing else. I’m not going to attempt to figure out actual performance times to search a table of that size, but it’s not insignificant, especially if millions of others are searching the same table. In fact, looking at how high those number get even with the current maximum of 5,000 friends, I think it’s safe to say that Facebook’s engineers are counting on the hope that 98% of their users will never create this many connections.

Now one argument could be that, even if Facebook doubled their maximum friends allowed, still only a VERY small percentage of their users would accumulate this many friends, thus keeping the database records to a much more manageable number. That said, Scoble once reported that they started to see scaling issues over 5,000 friends, so it’s entirely possible that they even this is not practical—I’m going to assume that the engineers know what they’re talking about.

Another problem is, even if we are certain that 98%(ish) of the userbase won’t connect to nearly this many people, there is still the latent possibility that they could. There’s also the possibility that a worm or script could do this for them, which is not as far-fetched as it sounds.

Without knowing the details of Facebook’s architecture, I’m still going to have to side in with Facebook’s decision—I’d say allowing more than 5,000 per person friends is probably a bad decision, architecturally. That’s not to say there’s no way to do it effectively, but I can see there being some big challenges. Sorry, Scoble. I think I’m with Facebook on this particular decision.

What Are We Building Again?

2008-03-29T05:46:00Z

The hardest single part of building a software system is deciding what to build. No part of the work so cripples the resulting system if done wrong. No other part is so difficult to rectify later.
- Fred Brooks

Some Things I Saw Today

2008-03-18T07:35:00Z

Idris is a dependently typed language. You can visit the page to see what that means.

Io is “a small, prototype-based programming language. The ideas in Io are mostly inspired by Smalltalk (all values are objects, all messages are dynamic), Self (prototype-based), NewtonScript (differential inheritance), Act1 (actors and futures for concurrency), LISP (code is a runtime inspectable/modifiable tree) and Lua (small, embeddable).”

Someone has written a Doom Roguelike.

You can see the slides to an SXSW presentation on designing interactions.

Ozimodo is a Tumblelog written with rails. A tumblelog is something like Anarchaia (from which I found a few of the above links.

Can’t think of a good name for that class? No problem. Visit classnamer.com. There you will be inspired to call it things like “PrioritizedGridDispatcher” or “GenericMemoryBundle”.

There is at least one website full of UML Jokes. Who knew?

Matz on Ruby 1.9

2008-03-07T12:00:00Z

Can’t leave out Matz; there are a lot of interesting things over in Google’s Tech Talks YouTube account.

Ola Bini Talks about JRuby

2008-03-06T14:19:00Z

The Tension Between Developing and Hacking

2008-03-04T18:31:00Z

Again, I’m going to continue to use “hack” in ye olde sense of interesting and/or excellent coding. And in this context, we’ll let “development” stand for the sort of coding you might do at work; something businessey, something enterprisey. It’s all good. I’ve been writing about this a lot. I hope that’s okay.

I wrote something called Considered harmful considered harmful not too long back, basically leapfrogging from a post _why made on hackety.org.

I saw an example today in the Ruby Programming Language book by Matz and David Flanagan that really makes it easier to get at the point I was trying to reach.

In the first chapter, they have a ruby Sudoku solving program, as an example of a little bit longer chunk of ruby code. I’m a sudoku nut, so I started picking it a part to see how it worked. Then I arrived at:

      # ...Writing dense code
      # like this is probably not good coding style, but it demonstrates
      # the power and expressiveness of the language.
      (0..8).collect{|r| @grid[r*9,9].pack('c9')}.join("\n").tr(BIN,ASCII)

(BIN and ASCII, in the above example, are string constants, ASCII = ".123456789" and BIN = "\000\001\002\003\004\005\006\007\010\011" respectively.)

Now, I’m not going to talk about how the snippet works — I’ll let you explore that on your own if you like that sort of thing. I was struck by the comment: Writing dense code like this is probably not good coding style, but it demonstrates the power and expressiveness of the language.

Yes it does. But that line makes very clear the tension between hacking (this is fun and interesting!) and developing (this is sober, and let’s make sure nothing clever works its way into the code).

In my mind, the comment read something like this:

LOOK AT THE REALLY COOL THINGS YOU COULD DO WITH RUBY. NOW, YOU ARE NOT ALLOWED TO EVER USE THIS IN REAL LIFE.

What?

Now, that’s just mean.

Installing JUnit 4.4 on Mac OS X

2008-03-03T06:17:00Z

“Installing” is a little misleading—since JUnit is a jar file, you aren’t really installing anything… you’re simply putting it somewhere.

The trick is to let java and your system know where to find JUnit.

So, let’s say you’ve downloaded JUnit 4.4 from http://junit.org. I downloaded the zip, so let’s assume you did that, too.

I put it in /Developer and ran:

unzip junit4.4.zip

Then, edit ~/.profile; you’ll need your path to know where junit is, and your classpath as well. My .profile wound up looking like this (only the $JUNIT_HOME stuff is really applicable, though you will definitely want to leave $PATH in there, so that you don’t get rid of everything that is in your path by default):

export JUNIT_HOME="/Developer/junit4.4" 
export PATH="/usr/local/bin:/usr/local/sbin:$PATH:$JUNIT_HOME" 
export CLASSPATH="$CLASSPATH:$JUNIT_HOME/junit-4.4.jar:$JUNIT_HOME"

To get bash to know about the environment changes you just told it to make, you could close and restart your terminal, or you could type

source ~/.profile

Now, you should be able to type:

java org.junit.runner.JUnitCore org.junit.tests.AllTests

and see results something like:

JUnit version 4.4
..........................................................
..........................................................
..........................................................
..........................................................
.......................................I.II...............
..........................................
Time: 4.098

OK (329 tests)

Rush -- a Ruby Shell

2008-02-28T22:52:00Z

Rush looks very cool. It is not “complete” but it is a usable shell, and is intended to replace bash & ssh with a shell that uses Ruby syntax.

I once thought about creating a Ruby shell (I think about a lot of things that I never actually do), but I was going to call it rash. It sounds more contagious.

UML For Hackers: Lean Class Diagrams

2008-02-27T15:53:00Z

UML is usually seen as a very corporate (i.e., boring) tool. It implies planning, best practices, and all those other things.

Hacking is usually described as a paradigm of just sitting down and starting to code, and altering your architecture along the way as you need to.

I think that’s fine, possibly even beneficial, for small projects, or projects worked on my one individual. But as soon as you decided to get a team working on a project… you need some way to represent parts of the application so people know how it works. Yes, we could just invent a new ad hoc representation scheme every time we need to do this… or we could just use UML.

UML has the advantage that some smart people put a lot of time into thinking about how it should work. Martin Fowler has a particularly good book on the subject, UML Distilled. This post might be thought of as UML Even More Drastically Distilled.

Class Diagrams

So, how about class diagrams, then. Here’s the bare minimum (IMHO):

Classes

A Class is a rectangle.
If you want to name it, name it at the top and put a line under it.
If it has attributes you want to remember, list them under the name.
If you want to add methods (class functions), draw a line under your attributes and list them there.
If you think some methods are obvious, then just don’t list them; as long as everyone else will think they’re obvious, too: get_x() and set_x() might be obvious. put_fuzzy_bunny_at_x() might not be.
You can use + or – in front of attributes or methods to indicate public or private respectively. (# for protected, if you want that).

Relationships

Subclassing (is-a) is represented by small open triangle under the superclass, and a line or lines to the subclass(es).
Aggregation (has-a) is represented by a small open diamond from “having” class, and a line to the attribute class. If another class is an attribute like this, you wouldn’t need to list it in the attributes necessarily; the class diagram makes it clear that Car has an Engine (for example).

There’s more, but do you really want to know it? We want a lean toolkit, here. Ok, fine; you can look at the quick reference card or this handy cheatsheet.

Computer Science vs. Real Life

2008-02-20T15:16:00Z

When I decided that I wanted to change my career and “go into computers”, I had a very limited exposure to computing.

Well… I should qualify that. It’s true, I had used DOS so much in high school (before there was Windows) that I preferred DOS to the Macintosh of that day, wrote my own autoexec.bat files, and in fact had a copy of DOS, Pascal, and Moria on 5.25” floppy disks that pretty much went everywhere with me. At the time, many DOS PCs did not have hard drives; you loaded the OS from a floppy, then it ran from memory and you put in some other disk with your files or other programs on it.

And I took a class called AP Computer Science which spanned 2 years; the first of which was all Pascal, which I found that I loved, and the second of which was all theory, during which I tuned out and drew cartoons in my notebooks (I was 17, what can I say?).

But after that… after high school, I really didn’t touch a computer for years. I didn’t own one, didn’t know many people who did. I didn’t really edge back toward computing until 1996 or 97, when I think I got my first email address (a Yahoo account which still accumulates spam).

So, when I decided to go into computers, it was by no means as a result of my casual familiarity with all things computer-ish. I had some distant background, but everything current was obscure to me. We had a PC at home which ran Window 95; and this was 2001.

I figured I’d need an edgemecation, so I looked at the curriculum of the local community colleges and saw “Computer Science.” That must be what I’m looking for, I thought, so off I went. In the meantime, I got my foot in the door by reading an A+ book cover-to-cover, getting a helpdesk job, and teaching myself to program on the web.

In the past few years of being in IT departments (both at small businesses and now at Oracle), in the various computer science classes at universities, and reading just about any and everything I can, I’ve noticed a few things.

For starters, as is well known if you are also in either computer science or software development, Computer Science as taught in schools does not seem to have much in common with “real world” software development. For this reason, you also see a major which overlaps somewhat with a Comp. Sci. major’s requirements: usually called MIS or CIS (Management/Computer Information Systems) or something along those lines.

It’s like Computer Science, only practical.

As it turns out, I’m more interested in Computer Science than Managing Information Systems, in general, so I think I made the right choice. But thinking of things like my past few articles, I’m trying to determine just why that is. (Though, honestly, “Managing Information Systems” just sounds incredibly boring.)

Interesting vs. Practical

Computer Science (my take, at least) is interested in structures and methods; what’s efficient, what’s interesting, why it works, and so forth.

More vocational programming (and hey: I am entirely in favor of developing software for a living) is much more concerned with processes, designing large systems, keeping code maintainable, being able to communicate what code is doing, UML diagrams, patterns, and so forth.

Now; I think there’s a lot of overlap there. But there are some big differences. Someone with a computer science mindset might be very interested in Lisp or Scheme (or Haskell, or Io, or Scala, or Ruby, or Python…) simply because of how they work, how they are different, and so on. A more pragmatic/vocational approach would have only one question: will anyone give me a job programming Lisp (or other from that list)? No? Okay; not interested.

In the second mindset, Ruby and Python only become interesting when you read that companies are actually using them, and paying people to write in them. In the first mindset, they are intrinsically interesting. Again, this is just a personal observation. It may be that most Computer Science students are really more interested in J2EE programming for Fortune 500 companies and are just taking Comp. Sci. because that is just how you do it.

But J2EE or whether we are interested in having good careers (of course we are) is not the point I’m trying to reach. The point, which in one way or another I’ve been writing about for the last few posts, is the dichotomy between the “clever hack” approach versus the “Great Wall Of Best Practices” approach, to programming in general.

Without re-treading too much old ground, one approach favors clever, neat, or elegant solutions. They might be shorter, or trickier, or more efficient, or merely unusual. The other approach favors… well, whatever is the least risky, basically.

I can understand the necessity of most best practices. If you are writing software that will be enormous, that will handle huge amounts of data, data which may be incredibly important to many people (people’s banking or medical records, or airplane routes, for example), there is a much greater amount of risk involved than if you are writing a new blog platform that you will use and frankly you don’t care if anyone else ever uses it or not. In the first case, failure of the software could be catastrophic to hundreds of thousands of people, up to and including loss of life; in the second, it’s a minor annoyance and you fix it when you have a chance.

So, if I’m a large company commissioning the creation of a huge, critical software project, I’m going to appoint someone to manage it, and I’m going to make it very clear that they are responsible for returning a product that works as advertised and will not wipe out everyone’s bank account or cause two planes to crash into each other. Oh, and also on time and under budget. User friendly? Only if you have time.

So now this senior project manager is going to have a team of people working on this software. She’s going to want batteries of tests run, procedures followed, and code that can be maintained. What if five of the best developers quit and become consultants for IBM? Well, someone had better be able to come in and continue working on their code, so it had better be readable, so do it like this....

...And we spiral into the process that, if successful, creates the Enterprise Software we know and loathe today.

And yet, I believe… I want to believe… that clever hacking can be kept alive and thriving in the midst of all this. I’m not sure that I can answer exactly how this can be done, but here’s the reason I think it can:

At the core, all of the problems faced by the development of Enterprise software, are still interesting problems, the kind that can engage someone in the clever hacks mindset. You need processes that allow code to be tested, maintained, that will be efficient, will allow for possible changing APIs of dependent software, changing platforms at some point in the future, changing file formats…? These are all potentially interesting problems. There have to be solutions that don’t involve sucking all the life out of the development process.

So even though the massive, test-driven, process-centric world of Enterprise software often seems at odds with the quick and dirty, use-the-language-of-your-choice, elegant-hack world of programming for fun, I don’t think it needs to be.

For an example (and maybe for a future post), I think this is part of the reason Rails has exploded in popularity over the last few years. It pleases (generally) both crowds. It incorporates all sorts of best practices and design patterns, while still being one big clever hack, using Ruby no less. I don’t think Rails is the be-all-end-all Ultimate Conclusion of this sort of thing, but I take it as a good example that, yes, you can please both camps.

Happy hacking.

Considered Harmful Considered Harmful

2008-02-19T04:08:00Z

Okay, the novelty of exploiting use-mention ambiguity wore off awhile ago. That’s okay.

It’s possibly time we stopped considering things harmful. The idea is somewhat puzzling; it seems as though the thing under consideration might not actually be recognized as harmful for any other reason than that we are considering it such. A panel of experts, perched somewhere behind your shoulder, has unanimously agreed that what you’re about to do may, in fact, be harmful. Dispensing with things such as proof, we’ll just consider it harmful and be done with it.

What are you doing? Don’t you know that’s considered harmful?

The trouble with this is that, to be consistent, we can’t even consider “considered harmful” harmful without violating our new rule. You see? That’s what happens when we start inventing considerations without regard to rhyme, reason, or reality.

On Iterative Development

2008-02-18T22:14:00Z

“You should use iterative development only on projects you want to succeed.”

- Martin Fowler (UML Distilled, 3rd Edition).

NOTE: I’ve had a penchant for just posting quotations, recently. This is mainly because I’ve been swamped with work, but part of that work has included reading or re-reading a lot of software development classics, hence the quotations. For what it’s worth, I’ll try to write something of my own some time soon.

Another Perspective On Advertisements on Blogs

2008-02-18T04:11:00Z

About 3 months ago I stopped running advertisements on my blog.

This was after about 2-3 years (not sure, really) of running AdSsense ads on my blog, and in that time earning a grand total of about $100. Needless to say, it wasn’t making much of an income.

During this time, I often found myself trying (usually in vain) to attract more traffic to the blog, but after awhile I began to feel like my motivations were wrong. The canonical strategy to advertisements on a blog, or almost any website, is that the more traffic you get, you should get correspondingly more click-throughs, resulting in more income. After awhile, my main goal was to simply make some money off the website. Obviously, there’s nothing wrong with this; I’m a big supporter of making money. But it seemed (to me, anyways) that my goal to drive traffic simply to make (hopefully) a few more dollars from the website, was distracting me from whatever purpose I had originally had in creating the website.

So I cut out the advertisements. Not a particularly difficult decision, they were bringing in practically nothing as it was.

Despite the fact that the net income from the ads was miniscule, I have definitely noticed that since I quit with the ads on the website, my motivation to bring traffic, and even to post regularly has subsided. I find this interesting, possibly a little embarrassing. After all, should I want to write simply for the sake of writing?

Well, not necessarily. Professional writers, for example, don’t write for free. In general, we are motivated by incentive of some sort in almost everything we do. So it’s not at all unreasonable that removing an incentive (advertising income… however small) would result in a decreased motivation to blog.

Because of this, I’ve been debating adding advertisements back to the blog again. My old blog and all it’s archives are still up (and getting substantially higher traffic than this new blog), so I decided to go ahead and put adsense ads back onto that site. After all, if the archives are just going to be sitting there getting traffic, I might as well have at least whatever slight income can be derived from that.

As far as this blog goes, the one that I’m writing now, I’m still undecided. Putting some sort of advertisements on the site would certainly reinstate some incentive to post more often. Would that incentive be offset by the temptation to drive traffic By Any Means Necessary, rather than just by writing the things that really interest me?

A sample ethical/pragmatic quandary: does having Amazon affiliate links make any potential book reviews I might write suspect? Less than honest? I can always determine to review only books that I love, but that’s not entirely honest either. If I read a book that really stinks (e.g., any of the Deitel series of programming books), it would seem a public service to share that with people before they make the mistake of buying it.

I suspect I’ll be pondering this for a bit yet. Since this blog receives such little traffic (currently, at least), it wouldn’t really matter if I added advertisements anyways. Still, I’m considering the pros and cons.

If you have strong feelings about advertisements on blogs, one way or the other, feel free to share them in a comment: thanks.

Abstraction

2008-02-18T02:08:00Z

“The discovery of common abstractions and mechanisms greatly facilitates our understanding of complex systems. For example, with just a few minutes of orientation, an experienced pilot can step into a multiengine jet aircraft he or she has never flown before and safely fly the vehicle. Having recognized the properties common to all such aircraft, such as the functioning of the rudder, ailerons, and throttle, the pilot primarily needs to learn what properties are unique to that particular aircraft. If the pilot already knows how to fly a given aircraft, it is far easier to learn how to fly a similar one.”

Booch, Grady, 2007. Object-Oriented Analysis and Design With Applications. Third Edition. Addison-Wesley.