john's blog

I occasionally get questioned about our choice to run our own network and servers when we could outsource it all to a "cloud" provider like Amazon and thus save a lot of money while simultaneously improving the reliability of our service. I hate this question because it is loaded, and I know they are not likely to understand why we can't run on a cloud service no matter how patiently I try to explain the technical issues.

Anyway, Amazon's EC2 cloud systems are now apparently dropping packets and having network latency issues. People running near real-time applications like gaming and VoIP are not having a happy time and they are apparently scrambling all around trying to figure out why their services are going to crap and, all the while, Amazon says there is no problem. I imagine that these folks did not get an SLA from Amazon with respect to network performance.

Whocouldaknown?

Link: Are spot instances killing the performance of Amazon EC2? - Seldo.com

I was sad to hear today that Tanta, a blogger at the finance and economics blog Calculated Risk, passed away. For me, the world will always divided into pre-Tanta and post-Tanta. She will forever mark a change in my reading habits. Outside of my narrow professional field, traditional media and journalism had always made up my primary source of information when trying to make sense of current events in the world around me. But that was before I stumbling onto one of Tanta's posts which shown bright and clear against the murky and frustrating background of noise being spewed forth by the usual suspects. From that day forward, I've looked for and found that the best signal to noise ratio on just about any topic can be had in the blogosphere. Thank you Tanta. RIP Tanta.

While a true story, the following contains what may very well be a bad idea...

I have an external 250 GB hard drive that, as of a few days ago, would not spin up. Upon power up, it would make an short buzzing sound following by, click, click, click, click, click and so on. I tried disconnecting and reconnected it a few dozen times with different cables, different ports, different power supply. No luck.

Finally, I decided to try freezing the drive by letting it sit on top of a large gel ice pack for 15 minutes. While still sitting on the ice pack, I powered it and it spun up on the first try! I then immediately began copying everything off the drive. The drive ran for about 3 hours while the ice pack slowly melted. Then it was back to click, click, click...

I got the idea from here:

http://www.macosxhints.com/article.php?story=2006110111270170

I don't currently have an official home office, but I've had an office phone at home for years now. This home office phone mirrors the phone on my desk at work. Both phones are registered as a contact for my SIP address so they both receive all my calls an otherwise behave exactly the same - all very easy to do with OnSIP Hosted PBX, but that's another story. A few weeks ago I swapped out my home office phone for an Aastra 57i CT which comes with this nice little cordless handset that syncs with the main station.

While it has never been an issue for me at the office as I'm pretty much at my desk most of the day, having the phone at home hardwired to my DSL router has always been a little bit annoying as I kinda like futzing around the kitchen and whatnot while talking. This is no longer an issue thanks to the cordless handset.

The main unit appears to be identical to the 57i and works as expected. Overall the Aastra cordless has been great, but I found installing the battery for the handset non-obvious - it took a little head scratching as to which way it was supposed to go in and I was also afraid that I might break the battery's connector/wire in the process. Another nit pick is that it took me a while to remember which button was to answer and which was to hangup. While the buttons are in logical places, they confused me for some vague reason and I found myself occasionally hitting the wrong one during the first week. But once I managed to get myself past those issues, it has been smooth sailing.

The upgrades are going well and we expect to be done around 2pm EST. Thank you for your consideration.

We are currently doing some network upgrades to add additional capacity to our core routers. While we don't expect any issues with 99% of calls during this period, it is possible that some customers may experience short term interruptions to some in progress calls. Thanks for your consideration.

I spent the good part of a couple of days last week helping track down packet loss at the edge of a PSTN carrier's IP network. They were dropping 1-5 packets out of every 10000 due to an incorrectly configured port on a gigabit ethernet switch. Now 0.05% packet loss doesn't seem like a lot, but at the end of the day it really depends on the phone you are using. If you are using a Polycom phone, you will likely never notice it. If you are using a Grandstream GXP-2000, the experience may be almost unbearable. It is my experience that the quality of IP phone software/hardware has by far the most significant effect on the end user's perception of call quality. And then there is packet loss. And sometimes the two go hand in hand.

Case in point is my hunt for missing packets last week. To review, the PSTN carrier in question was dropping 0.05% of the packets over the private interconnect my calls happened to be traveling. That's 5 out of 10000 packets. To put some context around that number, we need a bit more information. We were testing g711 at 20ms, so each packet contained 20ms of voice and thus we were sending 50 packets per second to carry one side of the voice conversation. That is, each packet represents 1/50 of a second of conversation and the carrier was losing up to 5 of these over 200 seconds. And those 5 lost packets were spread randomly across that 3+ minute period.

Now you might think someone would have a hard time hearing a 1/50 second gap in a 3 minute conversation. And you would be right. But if there is an IP network in the middle of the conversation, it turns out that drop detection depends greatly on the the piece of IP phone hardware/software one is talking on. In this case, the Polycom IP phone on my desk was useless as an empirical tool in helping me track down the 0.05% packet loss in question - I simply could not hear it. However, the Grandstream GXP-2000 (version 1.1.0.14) we have in our test bed turned out to be an invaluable tool since it has the uncanny ability to turn the 1/50 second gap caused by a single lost packet into a multi-second garbled mess. So, if like me, you enjoy testing for dropped packets, I highly recommend adding the Grandstream to your toolbox - it is a great tool and worth far more than the retail price.

On and off over the last couple of years, as Skype, YouTube, and more recently Facebook got tagged with silly sums, I've been having the occasional flashback to bubble-land circa 1999. The hype, the parties, and that good old frothy feeling. Ah, the memories.

Well, in case you hadn't noticed, eBay suffered "an impairment write-down" recently. In my opinion, Meg Whitman must have been impaired back in 2005 when she judged $2.6 billion to be a good deal for Skype. Skype is a great, disruptive, and revolutionary technology precisely because it doesn't extract much money from its users and it points the way to a time when communication will cost the consumer nothing at all. In the words of its co-founder Niklas Zennstrom, "We want to make as little money as possible per user." So it should come as no surprise that eBay said it would take a $1.4 billion charge last week. Just to get all the bad news out of the way in one fell swoop, eBay probably should have taken the kitchen sink approach and added on another billion or so to that number. eBay's board must be pondering the value of Ms. Whitman these days.

Well one thing is for sure, Mr. Zennstrom and friends look like some pretty smart folks at this point. And I can't help thinking about those YouTube guys too - I loved the "press release" video they put up with respect the ten figure number applied to their project. Those guys appeared so clearly stupefied and dumbfounded by the shear insanity of the valuation obtained. I couldn't help laughing with them. In Google's defense though, I suppose if I could print Google dollars I too would probably be willing to pay just about anything to get what I wanted. I wouldn't, however, say the same thing with respect to Microsoft.

eBay started the post-bubble bubble with Skype. I wonder how it will play out.

Have you ever wanted to hammer a part of your network or a particular system to get an empirical view of its performance characteristics under a VOIP load? Are you someone who is compelled to understand how things work in grim detail, even if the things in question are tedious in the extreme - like network performance measurement? If not, don't feel bad. As it was explained to me, "Not everyone who visits the blog is an UberNerd, or aspires to UberNerdity, but on the other hand those who display UberNerditude in the comment threads are treated with a respect bordering on lunacy. That’s just the way we are."

If you are an UberNerd like me, you may be interested in the topic of this post - a great little testing tool called Iperf which I recently had the opportunity to pull out of my UberNerd toolbox and use again.

A quick look at voip-info.org should make clear that designing even a simple VOIP platform can be something of a black art. This is particularly true once you step outside a single vendors sandbox. In designing any platform, an understanding of the characteristics and limits of the individual components that will make up the platform is critical. And while mathematics, physics, and white boards are the basic engineering tools in any such design endeavor, at the end of the day some real world testing and data collection can be invaluable - and given (in many cases) the lack of readily available pertinent data, this is particularly true when it comes to VOIP platform design.

When testing components of a VOIP platform, Iperf can be quite handy. Quoting from the Iperf documentation, "Iperf creates a constant bit rate UDP stream. This is a very artificial stream, similar to voice communication but not much else." While it operates in TCP mode by default, simply adding the UDP flag (-u) to all the commands will flip Iperf into a more suitable mode and adjusting the datagram size (-l) to 250 gives a close approximation to 20ms g711. Run Iperf on one system in server mode (-s) and on another in client mode (-c) and within seconds it starts spitting out nice reports on bandwidth, jitter, and loss. Finding the upper limit of system and network components is as simple turning up some of the Iperf dials (-b and -S among others) and waiting for jitter and dropped packets to show up. While it cannot provide the whole picture, combine Iperf data with the output of system monitoring tools like vmstat and an empirical baseline of how a system will behave in the face of VOIP traffic begins to materialize.

So I recommend considering Iperf next time you are adding to your UberNerd toolbox.

Giddy, strung out, and their schedules twisted more than normal, many of my associates are pulling all nighters. Every meeting I've attended recently has opened and closed with the same topic. In my small part of the world, the hottest topic in VOIP at the moment is Halo 3. I can't help wondering if I'm missing out.