Monday, June 18, 2012

Erlang Concurrency Model FTW?

While catching up on my reading list, I came across this performance benchmark of C10k web servers.
ImplementationConnection Time (mean)Latency (mean)MessagesConnectionsConnection Timeouts
Erlang (Cowboy)865ms17ms2849294100000
Haskell (Snap)168ms227ms11874134996108
Java (Webbit)567ms835ms10283904637157
Go284ms18503ms23981809775225
Node.js (websocket)768ms42580ms117084757014299
Python (ws4py)1561ms34889ms105299647925208
With the exception of Connection Time, Cowboy totally owns other web servers' asses in all other categories.

In the realm of high concurrency and availability, Erlang still rules supreme.

There are a couple of observations that I want to make:
  1. Despite all the hype, Node.js still fails to deliver. Don't get me wrong, I really like the idea of running JavaScript on the server. Most web developers already know JavaScript and can't be bothered to pick up a functional language just to write asynchronous and event-driven software, so the barrier to entry is significantly reduced. I just wish that all the attention and community participation would have resulted in something better in terms of scalability and performance.
  2. Although Cowboy's average Connection Time is second worst and eight times Snap's average, practically speaking most users won't even notice the difference as the former still takes less than a second. When it comes to performance, perceived responsiveness matters more than raw numbers. There's a big difference between comparing 100ms to 800ms and 1s to 8s.

Saturday, June 16, 2012

AWS Automation Using PowerShell - Part Un (Starting and Stopping EC2 Instances)

This is the first post in what will hopefully be a series of posts (good luck with that) on automating Amazon Web Services (AWS) deployments using PowerShell.


Pre-requisites


Before you get started, there are a few things that you need to do:
  1. Gain basic understanding of AWS especially on EC2, S3, EBS, EIP and SNS.
  2. Sign up for an AWS account.
  3. Install PowerShell 2.0.
  4. Install the latest version of the AWS SDK for .NET.
  5. Create a directory in which you're going to create the PowerShell scripts.
  6. Copy the AWSSDK.dll file from the bin directory of the SDK installation (e.g. C:\Program Files (x86)\AWS SDK for .NET\bin) to the new directory. This is to avoid having to deal with machine-specific installation paths and lets you deploy a self-contained set of automata. 
  7. Buckle up for a joy ride ahead!


AWS SDK for .NET

The SDK comes with a Visual Studio 2008/2010 plugin that lets you create .NET projects that you can then easily deploy to AWS using a wizard-based GUI. On top of that, it also includes a .NET library (duh) that wraps calls to the AWS REST APIs.

Having said that, the IDE plugin is pretty much useless for our purposes since a good engineer always attempts to automate repetitive tasks, and using a GUI-based tool does not a good automaton make. Instead, we want to be able to write a program, or better yet, a script that can be executed from the command line interface. You can choose to write the automata in C#, but bear in mind that they will ultimately be owned and maintained by a DevOps team. Most DevOps engineers I know can't be bothered to learn C#, so PowerShell is probably be a more preferred option.


Spinning EC2 instances up and down


While the best practice in managing an AWS infrastructure is to leverage CloudFormation and deployment platforms like Puppet or Chef, doing so requires significant investment in understanding the technology and underlying concepts. What I'm going to do here is present a simple scenario where you need to start and stop specific EC2 instances so that we can gain basic understanding of AWS and its APIs before delving into more advanced scenarios. Think of it as getting your feet wet before diving in head first.

The following is a sample PowerShell script that starts and stops EC2 instances. Bear in mind that I'm a relative virgin when it comes to Powershell, having only been to the second base. So keep the laughter and scorn to yourself for now.

The code is actually pretty self-explanatory, but I'm going to walk you through it anyway.
  1. The first line exposes four parameters, namely the operation (start or stop), the AWS account secret key, access key and a list of EC2 instance identifiers that you want to start or stop.
  2. Line 3 is where you include a reference to the standalone copy of the SDK.
  3. Lines 5-23 represent the meat of the script that starts and stops the instances. As you can probably gather, most AWS APIs follow the request-response pattern. You create a request to do something, send the request to AWS and get a response in return. It doesn't get any simpler than that.
  4. Lines 25-29 refer to a function that creates and returns a 'proxy' to the EC2 API based on the specified secret and access keys.
  5. Lines 31-41 contain a function that returns a generic list of EC2 instance identifiers based on the instanceIds parameter.
  6. Lines 43-50 refer to a simple switch that executes either the start or stop function based on the operation parameter passed into the script.
You can then use the script as follows:

Go to your AWS console and lo and behold, the instances are spinning up!







There are many ways of automating this process. One option is to create builds in your favorite continuous integration platform (TeamCity, Go, TFS, etc.) that make use of this script to spin the instance up or down based on a specific schedule. Alternatively, you can configure the builds to not execute automatically so that you can manually trigger them with a single click of a button.

What's next?

The next step is to bootstrap the EC2 instances during startup so that each instance is responsible for configuring and installing its own resources dynamically. We can do fancy things like leveraging the Simple Notification Service (SNS) to notify interested parties (e.g. the build workflow) when the bootstrapping process is complete instead of polling each EC2 instance to see if it's up and running. This and more will be covered in the next post.

Till next time, adios.

Thursday, June 14, 2012

Learn you some Sydney for great good!


From time to time I get questions from family and friends on pointers on what it's like to migrate to and live in Australia in general and Sydney in particular. After giving ad hoc advice and tips for quite some time, I decided to compile them into one nice package I'd like to call Learn you some Sydney for great good! If you or someone you know is thinking of migrating Down Under, I hope you'll find this handy.

Immigration and customs

Most expatriates arrive in Australia on one of the following visa classes:
  1. Temporary Long Stay Business visa (457) is a temporary visa where a company sponsors you to live and work in Australia for up to four years. Being on this visa also entitles your dependents to work and study (fees may apply in NSW and ACT) in Australia during your whole stay.
  2. Skilled Independent Migrant visa (175) is a permanent visa that allows you and your dependents to work, study and receive subsidized healthcare in Australia on a permanent basis. To be eligible for this visa, you have to have qualifications in at least one of the listed skilled occupations and be sufficiently proficient in English.
Australian Immigration uses an electronic visa system which means you don't have to have a physical visa page in your passport. To be on the safe side, it's probably worth having a copy of the e-visa ready in case the immigration officer asks for it.

If you are bringing anything organic like foodstuff and spices into Australia, make sure that you declare it in the entry form to avoid paying hefty fines. Australian Customs is very particular about these things.

Documentation

One of the first things that you should probably do as soon as you arrive in Australia is to get an official ID card. Doing so allows you to avoid the hassle of having to bring along your passport, utility bills, bank statement, etc. to meet the 100 point check requirement to prove your identity when opening a bank account, applying for a bank loan or signing up for mobile or broadband services.

There are a couple options, both of which meet the 100 point check requirement:
  1. If you have a valid US (or any OECD country) driver's license, you can simply convert it to a NSW license.
  2. If you don't have a valid license from any of the recognized countries, things can be a bit tricky. I recommend opening a bank account and getting your first utility bill (as proof of address) before applying for a photo ID card.
Banking

The retail banking system in Australia is probably a bit more advanced than what you're used to in the US. For instance, most people don't really use checks anymore, but then again that's not saying much.

Most new expatriates in Australia tend to go with any of the big four banks for their financial needs. Personally, I prefer HSBC since I frequently repatriate funds back to Malaysia and having an HSBC account back home makes things much more convenient. HSBC also charges a relatively low fee ($8) for making global transfers and offers above average fixed deposit rates. Better yet, if you're an HSBC Premier account holder, you can even make global transfers for free and take advantage of a number of perks that they offer like being able to open an Australian account before you even arrive in Australia and a free Qantas club membership (totally worth it if you fly a lot), among other things.

Housing

This is probably the single biggest thing that's going to hit your pocket, especially if you have a family. Suffice it to say, housing here is among the most expensive in the world. Heck, even New York rents are considerably cheaper in comparison. Be prepared to fork out close to $700/week for an unfurnished 3-bedroom apartment in metropolitan Sydney. While you still can find fully furnished 4-bedroom houses in suburban Sydney for $400-500/week (which is considered really cheap by the way), bear in mind that anything beyond 20km radius is considered somewhat dodgy or ghetto here, at least in my opinion.

When house or apartment hunting, your best friend is the Domain website. It allows you to customize the search by specifying a comprehensive set of criteria such as whether the property has broadband access, built-in laundry or dishwasher and proximity to schools, shopping malls and train stations.

The North and North West suburbs are among the best places to live in Sydney. I prefer the North Shore due to its family friendliness, the excellent accessibility in terms of public transports, availability of all sorts of amenities and good schools, proximity to so many beautiful and pristine beaches, as well as its more affluent demographics, although the North West suburbs are not so bad either. Bear in mind that because of the aforementioned factors, properties on the North Shore are more expensive than those in the North West.

One of the best places to I live in Sydney is a suburb called Chatswood located 12km from the CBD. It's a lovely neighborhood with a lot of shopping malls, schools and a transport hub. On average, you only need to walk 5-10 minutes in any direction be it when grocery shopping, walking your kids to school or catching the train to go to work.

Another decent suburb is Rhodes. Located 15km from the city center, it's a fantastic bayside suburb with has recently been re-developed and boasts beautiful and modern low-rise apartments, a shopping mall, restaurants, IKEA (you can literally buy furniture and drag it across the street to your apartment) and a train station. There's also a public school about 15 minutes by foot and a few other decent schools a mere couple of train stops away in Strathfield. It's perfect for families who prefer someplace more quiet and want to avoid the hustle and bustle of Chatswood.

Getting around

Depending on where you live, public transport can go from excellent to 'meh'. In general, if you live in Sydney metro, chances are there's a train station a mere 5-10 minutes walk away. However, it's worth noting that some areas in Sydney metro are not covered by the rail network especially those that are close to the beaches, but even then buses frequently ply those routes.

If you decide to live in the suburbs, then you'll probably need a car. In general, cars here are a bit more expensive compared to what you'll find in the US. For instance, a brand new minivan or full-size sedan will probably set you back $35,000-45,000. Gasoline and maintenance are also more expensive thanks to higher labor costs.

That's exactly why I decided to live in metropolitan Sydney since everywhere is within walking distance. In the rare occasions when I need to use a car (like driving to the beach or countryside), there are car sharing services available. Just pay $9-29/month for membership and $5-8/hour plus mileage every time you drive one of their cars. I find it to be extremely convenient and hassle free, not to mention cost effective.

Healthcare

If you're a permanent resident, then you're entitled to the publicly funded universal health care system (Medicare). Bear in mind that you need to spend roughly 2% of your income on the Medicare levy to take advantage of it. However, you can avoid paying the levy if you sign up for a private health insurance. In that case, I recommend NIB as I know a number of friends who have been with them for many years and are very happy with their service.

If you're not a permanent resident, then you'll need to sign up for a private health insurance as part of your visa requirement. Depending on the package (basic to comprehensive), a typical health plan will probably set you back $300-500/month. I recommend IMAN which is a subsidiary of NIB that offers services to expatriates on working visas.

I'd say Medicare already provides fairly comprehensive coverage for most families. Bear in mind that there are a number of things that are not covered such as dental treatment, drugs, elective procedures, etc. In addition, Medicare typically only covers 70-80% of the costs for specialist consultation.

Having said that, even if you decide to sign up for a private health insurance, there are usually waiting periods of 6-12 months for extras and pre-existing conditions.

Schooling

If you're a permanent resident, then you get to send your kids to public schools for free. You'll still need to pay for stationery, school uniforms and excursions though, but the costs involved are nominal compared to what your kids get in return.

In general the quality of public education in Sydney is pretty high. You don't need to send your kids to a private school to get quality education as the public schools here are just as good.

If you're into numbers, you can find a number of school rankings here. There's also a good website that profiles each school and provides comprehensive statistics on their academic performance.

Among the public primary schools in Sydney, Artarmon Public School is probably the best, while Chatswood is in the top 10. Hornsby North and Kirribilli aren't too bad either.

If you're not a permanent resident, then be prepared to spend $5,000/year per child to send her to a public school. If that's the case, then you probably want to consider sending your kids to a private Islamic or non-denominational school, some of which only charge $3,000-4,000/year.

Basic Islamic education is taught in public schools under the banner 'Scripture', although the availability depends on whether there are enough Muslim students there. If your kids require something more in-depth, there are a number of private Islamic schools in Sydney, although unfortunately most of them are situated in suburbs like Bankstown and Auburn which I consider to be somewhat dodgy.

Halal food and mosques

Halal food is really easy to find here. Most kebab parlors, Nando's and Indian restaurants, even some Oportos and KFCs serve halal meat. Just make sure to ask first if you're in doubt.

If you need to buy fresh halal meat, there are many halal butchers in Sydney. If you're feeling lazy and want the meat delivered to your door instead, check out this butcher. I usually go to a butcher in Asquith, a suburb 30km from Sydney CBD but is still convenient nonetheless since it's located just next to the Asquith train station.

Mosques and musallahs are also easy to find. If you work in the CBD, there are a few mosques that conduct Friday prayers. There's also a mosque in Artarmon close to where I live that conducts Friday and Eid prayers and has an active Muslim community.

Telecommunication

When it comes to telecommunication, sadly things here aren't much better than in the US.

Mobile plan: Just skip the rest and go with Telstra. They are hands down the only reliable provider here. That said, reliability comes at a price, so you'll probably end up paying a bit more for a Telstra mobile plan, although the price gap is getting smaller by the day. If you already have a phone, just sign up for the $30 pre-paid plan that gives you 2 hours worth of talk time and a 'whopping' 500MB worth of data! Oh, and be prepared to be baffled by the pricing and marketing BS that's designed to confuse consumers. Just read between the lines and you'll be fine :).

Broadband: If broadband access were the only barometer for how developed a country is, then Australia would probably end up in the third world category. Most ISPs impose monthly bandwidth caps on broadband usage (typically 50-500GB/month), although some don't. Personally I find TPG to offer the best value for money. I currently pay $30/month for an unlimited ADSL2+ (up to 20Mbps) package and am very happy with it.

The Australian government is in the midst of deploying a nationwide fiber-to-the-home (FTTH) infrastructure which promises to significantly improve the quality of broadband here. At the moment, the coverage is still few and far between and pricing somewhat unaffordable, although things will definitely improve over time.

Television: If you're NOT into sports, then you can just make do with digital free-to-air TV that offers a number of HD channels and quality programming (minus the lame reality shows). If you ARE into sports, then you really only have one option: Foxtel. They have total monopoly over paid TV here in Australia, so be prepared to spend around $100/month on a HD sports package.

Safety and security

In general, Sydney is a very safe and family-friendly place. In fact, Aussies are actually a very friendly and welcoming bunch, and there's a good mix of people from all sorts of ethnic groups living in harmony without any problem.

That said, it's still probably wise to avoid some of the dodgy suburbs, especially those in the inner west and south west sides of Sydney. Also, try to avoid the CBD and areas close to pubs on Friday and Saturday nights as some Aussies can be somewhat unruly when they're intoxicated.

Expenditure

The following table contains sample monthly expenditure for a family of four (2 adults and 2 children) in Sydney whose main provider is on a working visa (457). All figures are in AUD.


Rent (2 bedroom apartment)
$2,500


Utilities

Water
$0
Gas
$50
Electricity
$50
Mobile pre-paid
$30
Fixed line broadband
$60
Pay TV
$80


Private health insurance
$400


Transportation

Car loan
$600
Insurance
$90
License and road tax
$45
Petrol
$200
6-monthly service
$50
Repairs
$20
Public transport
$150


Groceries
$800
Eating out/lunches
$200
Schooling
$400
Recreation
$150


Things to do

You may feel somewhat overwhelmed when you first set foot in Australia. It's normal for new expatriates to find it to be very daunting and confusing to settle down initially. There are so many things to figure out and tasks to juggle. For the sake of simplicity, I'd prioritize the tasks in the following order:
  1. Open a bank account
  2. Sign up for a mobile plan
  3. Medicare registration
  4. Apply for a tax file number
  5. Get a photo card or driver's license
  6. Attend job interviews
  7. Go apartment hunting
  8. School registration
  9. Get a broadband connection
Getting an ATM or debit card will earn you 20-30 points (towards the 100 point check), so that's definitely something that should happen on the first day if possible. Then you'll need a mobile number so that the authorities, landlord, service providers, etc can contact you.

You should also consider getting a job first before getting an apartment as most landlords and real estate agents require proof of income before they can even consider your application. You can start applying for jobs while still overseas and arrange for interviews when you're finally in Sydney.

Tuesday, June 12, 2012

The Final Question

This is an old short story that I wrote 10 years ago. I just realized today that I left it unpublished.

The Final Question


She felt rather uneasy. In spite of the discomposure, she sat very still in the car seat, apparently still bothered by the question. He was driving quietly without even having the radio turned on. It was unusual, awkward at best. She slowly sank deeper into the seat, still without a word to say.

In the aftermath of the question, everything went silent.

Until moments later. The car shuddered momentarily as it was going over a rough ground. The change was too abrupt for him to react to. He was driving a bit too fast despite the peculiarly dark and bleak evening. She was tempted to bitch about it, but decided against it this time. He was anticipating her bitching as usual, but it never came.

And the previously enduring silence continued.

The streets were hollow, their inhabitants idle. Amidst the utter stillness, he still managed to catch a glimpse of a stray English beagle from the corner of his eye. It was apparently looking for food atop a massive pile of garbage. The sight was saddening and disturbing at the same time. Saddening because of the fact that such an adorable breed was left astray. Disturbing because of the existence of such humongous pile of rubbish in the middle of the metropolitan area. And the silence between the two of them only added a melancholic air that began to invade the tiny space in the car.

On the far horizon ahead, a set of bluish lights appeared. Moving closer to the source, they could discern a congregated group of vehicles in the middle of the road. A bunch of heavily built figures were standing alongside the cars. They looked at each other silently for the first time since the question popped up, albeit lackadaisically.

It was the police. He quickly slowed down his car to a complete stop.

Rolling down his window, he could see a menacing look on the officer's face. The officer gave him a brief but calculating stare, obviously not pleased with what he was seeing.

“Good evening officer,” he braved himself by greeting the copper.

The officer only half-nodded in reply, doing his best to appear grim and merciless. Then the cop turned to look at the passenger seat. Seeing the sweet but deliberate smiling face of hers, the officer grinned sheepishly. The officer then turned to look back at him, but the grin was immediately gone.

“Bugger off,'” the officer said, giving him the thumb.

“Thank you,” he replied mendaciously, unhastily hitting the pedal.

Bloody bastard.

He drove on. She was still in the very same sitting position as she was a half hour ago, still lost for words. Such was the compelling outcome of the question brought up by him. It stood to reason that he was biding time, devoid of haste. Let her think, the tiny voice inside of him commanded.

Five minutes later, he saw the usual signboard that read “MANHEIM - 15 miles”. For some strange reason, his feelings were suddenly overcome by sadness. It's close now, he thought to himself. She still had not uttered a single word. It was agony, a miniscule and insignificant portion of what hell was all about.

Still the deafening silence of self-repression persisted.

But he couldn't stand it anymore. His old man was right all along; silence kills. He wanted to turn on the radio, hesitated for a moment, then finally went ahead with it. He cast a furtive glance at her, expecting a response to his shattering the silence. But she just stood there soundlessly, like a frozen cryogenized figure, probably deep in thought.

The soothing voice of a female singer came out of the front and rear speakers. It sounded familiar indeed, but he couldn't quite make out who it was. The tempting thoughts of asking her lingered in the air, but he couldn't bring himself to commit it. He was about to open his mouth to ask, but his tongue was completely numb. Sheryl Crowe maybe, he speculated to himself instead.

They were five minutes from the destination. He deliberately took a left turn at one corner, buying more time by following a longer route. The melancholic air was getting thicker and denser, as they were approaching in on the destination. He could faintly hear her weep, but stopped short of glancing at her. Deep in sorrow, his breath got heavier, as the Victorian mansion was now visible around the secluded corner of the neighborhood.

As they arrived in front of the mansion, he stopped the car, turned off the engine and slowly turned to look at her. She was already facing him, looking surprisingly calm and quite at ease. An awkward moment of silence ensued, as he had a rather expectant look on his face. This is it, he thought to himself.

Then she took a deep breath, and deliberately said, “Yes”.

She then paused for a brief moment, quietly but calmly gasping for air. The answer sent a wave of strange sensation to his heart, then slowly down his stomach.

“I still do,” she added.

There was something about the way she uttered the words. It was rather reassuring. Finally, she kissed him gently on the cheek and immediately got out of the car, apparently teary-eyed and not wanting him to notice. He watched her as she walked slowly towards the front door, and into the mansion. She never looked back.

He left after a passing moment of contemplation. Driving home serenely from her house that night, he still managed to form a smile, even though he realized that he wouldn't see her ever again.