Best MMO Item

Interesting thread on the FOH message boards with people listing their favorite MMO weapon and item. Steve Danuser (Mobhunter) has some good analysis on the reason EQ items dominated (then and now) that I think is spot on. FWIW, my favorite weapon was the Staff of the Wheel (wizard quested item) and my favorite item was one of GBS, robe dropped by Najena or Jboots.

Python, Closures, map and filter

Closures are probably one of the greatest tools a dev can have in his toolbox. I used to use them a lot when I did assembly programming where they were called "macros". According to Martin Fowler they were called "Blocks" in Smalltalk. Ruby has them, Lisp is pretty much nothing but closures. They're freaking great. I mean really, really, really freaking great. If you don't use them you should learn how. Like now. Really. Unfortunately, if you're a Java programmer you're pretty much out of luck. I remember seeing a write up about somehow, kinda, sorta making closures work in Java but it seemed like a clusterfuck. But whatever. Just start using python (or maybe jython). They come with the box and they're drop dead easy to use. If you want some examples see this: click me

With closures map() and filter() become nectar of the gods. This is the magic combination:

closure+map|filter|<other-iterator-func>

With this combination all kinds of code can be squished down into near non-existance. I like to think of it as the neutron star of python: compacting code down to miniscule levels. It's not everyday that I come across a case that needs the neutron star pattern but when I do the day always seems a little brighter.

Yearning to Game

I've played a lot (A LOT) of games in my life. Board games, card games, video arcade, consoles, rpgs, mmos, you name it. There are two that stand out in my mind: Ultima Online and Everquest.

I started playing Ultima Online in summer of 1997. When I first heard about UO I poo-pooed it. But as it got closer to release I started reading more about it and it sounded just incredible. I really wanted to be in the beta but the due date for sign ups had long past. So one day I'm telling my buddy Jim Cullen about it and he says, "oh yeah, I signed you up for the beta like a year ago." Huh?! Surer than hell when beta started I got a shiney CD in the mail. Holy shit! I could have kissed that guy.

My first experience with UO was a lesson in patience. The servers would crash constantly (like every hour or less) and the lag would often be horrendous. But let me tell you -- I did not give a shit. I LOVED IT!! I created a newbie and found myself on the docks in Vesper. I was in heaven. I proceeded to waste the entire summer of 1997 exploring Britannia. Release came at the end of September and for roughly 2 years I canvased the land learning all of the ins-and-outs of the game, amassing a fortune, PKing the poor souls who crossed my path and generally having a blast. I had some great friends (CoconutMonkey, Desmodus/Batticus, Uber and host of others I can't remember). If you're reading this and want to know if you ran into me I played on Atlantic and had the following characters: Sunchaser, Kafka (PK), Lia. If I PKed you, well, sorry. But rest assured I took a few hits myself.

I've gone back to UO a few times since I left. Unfortunately, I sold my user account so I could never revist my old characters. But I did enjoy once again running from Vesper to Minoc to Britain and down to Trinsic. The music still haunts me. Last time I logged in, perhaps a year ago, I was shocked to see the game was still laggy. 10 years and still they hadn't fixed the lag. That's the software biz for you, I guess.

Everquest was my second mmo and real love. I had actually been in the EQ beta but I wasn't sure it was for me. I started working at HomeAccount in summer 1999 and a friend of mine, Curtis, had been playing since release. I would often go over to his desk and see what new places he had visted and admire the great trinkets he had picked up along the way. So one night I called my friend Deon and convinced him to ride to the store with me to pickup the game. The next day or so my friend Kevin also bought the game and the three of us spent about 3 months adventuring all over the place. At around level 30 my friend Deon bid us farewell but Kevin and I and my friend Jim (living in Detroit at the time) kept at it until we reached the "end game" content.

If you've only played WoW but never EQ then you can't really appreciate the level of dedication that goes into getting a character from 1 to 50 during the early years of EQ. We're not talking a little work here. We're talking a freaking fulltime job. You could have easily spent 8 hours per day, everyday working on your character and would still take you the better part of a year to reach the end game. But once you got there it was a different game. Everyone had gone through the same shit you had gone through. There were always the uber-guild clicks that exluded everyone (I guess that was the thing to do) but most people appreciated their fellow masochists and gladly welcomed them to a party in Lower Guk, Sol B or the occassional Vox or Naggy raid.

I guess I played EQ off and on until WoW was released. So roughly August 1999 to November 2004. I still think about how much I enjoyed my time in EQ. I think I actually hated-loved every minute. I so wanted to rocket up to uber but I would spend a week (or more) learning every nook and cranny of a zone. I guess what I did in EQ is what I did in UO: enjoyed the journey. WoW, bah, "let's just level already".

So there is all of that. Now I'm tinkering around with Project EQ. It's basically old school EQ using the EQemu emulator. I'm curious to see how long I last before I toss up my hands in frustration.

Twitter, FriendFeed and Streams

I like Twitter. The signal-to-noise ratio can be low sometimes but there are work arounds for that (eg filter tweets through friendfeed). More people need to use it and the people that do use it need to do so more often -- well, except for @timoreilly who i'm convinced is actually a front for a team of overworked researchers.

FriendFeed is nice too. I like to think of it as Twitter on steroids. I'm in the process of reshaping the way data flows into my feeds there. For example, I want lots of data from devs. Amitp is a fantastic dev and any insight I can gain into the way he views the world invariably helps me. There are other devs I follow there for similar reasons. One problem I do see is that I may have a problem filtering out enough information. For example, if someone I'm following comments on (or likes) someone else's post then I get the post and (maybe) their comment. Often the post is not something I would be interested in. It would be nice if I could somehow massage my feeds so that this extraneous data is invisible. It could be that FriendFeed has facilities for this already and I haven't found them. I'll look into that.

The emerging pattern here is that more and more I find myself in the middle of streams of data. FriendFeed and Twitter, yes, but also Reddit, Slashdot, Facebook, all of the feeds running into Google Reader, Blue Yak message boards, etc. These things can suck up the better part of the afternoon during which time I'm doing no dev or dev research. I'm going to need to 1) schedule times for stream reading and 2) better manage/filter my streams. A social rating/ranking system would be nice. "Here is a stream of data rated for you." Is that what FriendFeed is up to? Probably.

Social Graphs Gone Wild

Social graphs have been around for a while. IM buddies, email lists and address books (old school black books) are forms of social graphs. While the idea is not new only recently have Myspace and Facebook brought the social graph into the larger public consciousness (Friendster and a few others were around even before them). Today there are many apps that try to tap into the "power" of the social graph: Twitter, FriendFeed, Orkut, etc. My goal is to figure out the best combination of these apps to "improve" my life. It's the entire thing isn't a net plus than I intend to dump. My current setup:

Facebook (profile pretty well filled out, modest number of friends added)
Twitter (integrated into Facebook now so will update more often)
FriendFeed (pulling in Flickr, Twitter, Delicious and YouTube)

Not yet integrated:
Blue Yak (group and message board)
AIM/ICQ
Email (this one seems iffy)

I'm sure I'm missing a few. I'll cogitate on this and write up my analysis.

Visual TODO: Organize the Garage

My first visual todo task is to organize the garage. I have family coming for Thanksgiving dinner and I would like for them to be able to navigate the garage without fear of getting lost (it's pretty bad in there). Images coming soon.

(20 minutes later) Images have arrived:



It's a mess in there. Let's see how this goes.

And 11 hours later... I didn't get to spend as much time in the garage as I would like but I managed to square away about 30-40% of the task. Here is the before and after:



A few things I need to consider:
- hooks for hanging the kayaks and bikes
- a bit more wire shelving near the freezer

The Visual TODO

Note: This is little more than a brain dump at the moment. I have mulled it over a bit so it's beyond the "brainstorm" phase but it is far from a polished idea. I think it's a great idea so I want to post it up (even as raw as it is) so that I begin testing it.

I'm going to try a new strategy designed to make me more productive (especially with regard to house chores). I'm going to call it "visual todo". The idea is that I will integrate my camera into my todo list. First I will take "before" photos of whatever it is that I'm placing on my todo list, begin work on the task and take "after" photos once complete. If I'm unable to complete the task in a short period of time then I will take "status" photos instead. I also intend to ball up the entire process as a dumpstuffhere blog post so that I can better track what I am doing and (just as importantly) what I should doing. Let's see how it goes.

Reading Shizzle

Fucking Yegge is right. As a developer you really need to be able to speed read. I totally suck at that. I gotta figure that one out somehow. I'd say I'm running at somewhere between 40-70% capacity in that area. Typing... "I got that shit down." But. Like. My. Reading. Skills. Really. Bite. So how? I read tons but it never seems like enough or fast enough. And I totally read the wrong things: junk food from reddit, political rags, a few decent blogs. What I should be reading are good dev groups, blogs. A good python resource would be nice. A sqlalchemy group. But still... faster... how? Stupid brain.

Google App Engine Followup

Following up my last rambling post with some GAE deficiencies that I have found.

Inability to communicate with external servers in meaningful ways

Often times when building an application it is necessary to communicate with another server. A classic example would be an email server. While the client connects to CRUD email the server would need to connect to other servers to deliver new email. Granted, you can use GAE's facilities to do this for you but if you wanted to invent "Email2.0" and need to directly poll servers or otherwise communicate with them you can pretty much fuggetaboutit.

That being said, you can make HTTP/S connections (80/443 only). However these connections have to be made within the context of a user request to the server (youch!). Worse, you can't time out these connections. So you could potentially hang your user indefinitely without any abiltity to control this from the GAE side. So, these just are not useful. I'm not sure why they have limited connections in this way. I'm guessing security concerns.

Cost of running routine database maintenance seems very high

I'm not a rocket scientist (OK, you didn't need that pointed out for you :P). It is very difficult for me to get an app up and running well. My preference is to build, tinker, build, tinker, etc. Some people like to start with requirements, move on to use cases, create a design doc and on until they have an app ready for the world. I prefer to start small, move it into production and continually add. That's just "the way I roll." That being said I find that the GFS/Big Table/Datastore on GAE doesn't fit well into the way I work. For example, API calls are supposed to be scarce. For the free/beta version you are limited to 2.5m API calls/day. That seems like a lot but I will often build some type of monitor utility that will continually scan my db and do things like massage rankings, update FK-relationships and various other tasks. In my apache+postgres/mysql world that isn't a big deal. When load is light I can fire off the maintenance apps and they can have their way with the db. No can do with GAE. And this is for 2 reasons:

1. There is no way to run a chron job on GAE. Worse, even if you could fire off a simulated job by touching a URL the engine will limit your processing time to that which is reasonable for a typical user request. Probably < 1 min but definitely < 5 mins.
2. Maintenance utils would burn through the API credits. Let's say I have a modest app with 100k records that I massage every hour or so with a ranking. 100k * 24h = 2.4m api calls per day and my users haven't even done anything yet.

There are some ways that I could work within the constraints of the app engine but for me, for now, it makes more sense to stick with LAMP on some rented hardware. I'll check back again in a few months and see if any of this gets addressed.

GWT notes

On the plus side I was able to get GWT working with GAE rather effortlessly. Pretty straightforward if you don't mind rolling your own RPCs. I think the RPCs out of the box are pretty bloated anyway. FWIW, I still find it strange that Google picked Java over Python for GWT. They should just buy out the Pyjamas guys and integrate it already.

Google App Engine

I've been tinkering around with Google App Engine (GAE) and I'm pretty impressed. It's clearly designed to compete with Amazon's Elastic Compute Cloud (EC2) and hopefully we'll see even more scalable app containers come on the scene. I kind of wonder if BEA will get into the mix anytime soon.

First, the good.

Getting a hello-world app up and running on GAE is a breeze. You can download the SDK, follow their tutorial and have something going in just a few minutes. Even growing your data model is a dream. Just create a class extending one of their datastore base classes (eg db.Model), use or extend the ready-made datastore properties and that's it. No worrying about mappings, creating tables, etc. All of that is handled automagically. The way the data models work seems very much like Django (which, apparently runs like a champ inside of GAE).

Most of the Python 2.5 library is available. A few things are missing (namely file/system-related libs) but I haven't really run into problems with that yet. I could see where it might be a pain in the ass later but hopefully I can just code around those problems or delegate out to EC2 or some other host. That would be ideal. I haven't really tested the interoperability yet but there appears to be at least a way to another server via http.

On to the bad.

I don't have a whole heck of a lot to add here (at least not yet). One thing that is a little difficult for me is the way relationships are handled. It's a little different from what I'm used to. It's sort of possible to model things like they would be done for an RDBMS but it definitely cuts across the grain. Even using the "officially-sanctioned" methods of dealing with relationships seems to be a bit problematic. For example, I would like to create something like a dicussion board. But how would that be handled. I'm thinking along these lines:

Board has many Forumss has many Threadss has many Posts

That's a pretty straightforward model. But when I start calculating datastore calls (because you are limited here) I get something like this (guessing numbers):

• select Board (1)
• select Forum (10)
• select Thread (25)
• select Post (15)

So just a single pass for a single user to view the posts in a single thread would generate 1+10+25+15=51 datastore calls. Figure that the average user probably clicks on what 5 or 10 threads? Let's just say 10. So that's 510 datastore calls burned per user for the dicussion board. We have 2.5 million datastore APIs we can use per day so 2.5 million / 510 = 4902 users we can support per day.

That's plenty of room for a pet-project, just-my-buddies kind of application but if you're thinking bigger then you're going to fall down pretty fast. Of course when GAE goes live you can pay to go beyond 2.5m API calls but I think the best approach is to rethink the design so that less calls are required. Having some kind of custom-built cache that works well with the datastore (which I think is BigTable underneath) is probably the best bet. Let's consider an approach where we cache all (10) forums into a single call, all (25) threads into a single call and all (15) posts into one more call. So now we're down to just 3 API calls (the initial board call would just call our first 10-forum block) * 10 interesting threads so roughly 30 API calls per uesr. So now we could potentially serve 2.5m/30 = 83,333 customers served. Now we're talking! Factor in memcache and you could see even more improvement.

So back to "the bad". The reason this is bad is that I really haven't thought about caches like this before. So I find myself optimizing earlier than I should be out of fear of scalability problems. OK, so that's probably a limitation of me and not GAE.

Why Blog?

I've had several blogs over the years. I even had one before they were called blogs (we just called them "pages" then). So why-the-frig am I starting a new one? There are several reasons. I'll list a few:

• Sometimes i just want to write a note to myself and a blog is good place to do that.
• Writing is a skill and I do not do enough of it lately. My hope is that I'll come here often and dump/organize/reconsider ideas and in the process improve my general as well as developer-related writing skills (yeah, good luck with that).
• Why not? I mean it's so freaking easy to do these days. I setup up this Blogger account in like zero minutes.

Another question I have for myself is, "who is my audience?" Just me, I'm thinking. No one is going to read the drivel I put in here. There are lots of places for devs to read other real devs (I suck, my skills are weak, such is life). Here are some I read:

• Zanshin.net
• Amit's Thoughts
• Stevey's Blog Rants

I guess I read some others but I can't think of them at the moment. It's times like this I wish I had some switches in my brain where, you know, it would actually work. Sheesh.

So that's it. First post done. I'll add another in a few but this pretty much sums up the beginning of this here place.