An unexpected gotcha in ZooKeeper watches
I’ve been doing some work with ZooKeeper at work lately. Specifically, we’re trying to use it to manage cluster topology (this work is related to my previous post on Berkeley DB Java Edition). Using the ZooKeeper Java bindings, it looks like the watch functionality is a beautiful thing. It is, actually…however there is a gotcha.
ZooKeeper will let you watch any znode for one of four events: created, deleted, data changed and children changed. These four events encompass everything you would ever want to know about a znode. These events aren’t where the gotcha lies. The gotcha lies in when they are fired.
The created and deleted events are fired (predictably) whenever a watched znode is created or deleted. Data changed is fired whenever a watched node’s data is changed. Children changed is fired whenever the children of a watched node are changed. This seems like it would meet all your needs, unless…
Let’s say you have a znode named “/members” to represent the members of a cluster. Each member creates an ephemeral node “/members/member-N”. It would be useful to clients wishing to communicate with the cluster to be alerted anytime a member is added to or removed from /members. So you subscribe a watcher to /members. When /member/member-1 joins the group, your code watches for the created event…which never comes. Likewise, when /member/member-1 leaves the group, your code watches for the deleted event…which never comes. ZooKeeper doesn’t roll like that. Instead, ZooKeeper fires only the children changed event.
This seems all well and fine, you’ve been notified that a change has occurred. Unfortunately you don’t know if it’s a creation or a deletion that has triggered the children changed event. If you’re using an internal data structure to represent the current state of the cluster, you don’t know what path needs to be added to or removed from your internal representation of the cluster state. Instead, you must get the entire list of children for /members and update your entire internal state (either rewriting it entirely or doing a diff and update).
Anyway, this isn’t earth shattering and it’s not a problem that can’t be solved. But one day I spun my cycles for a bit wondering why I wasn’t seeing any creation or deletion events for children of a watched znode and I didn’t immediately know why. So I figured I’d share this and (hopefully) save someone some time.
For more information on ZooKeeper watches, read Animesh Kumar’s very nice writeup.
Berkeley DB Java Edition High Availability
It’s been over a year and a half since my last real blog post. What better way to get back into the swing of things than a technical post about some things I’ve been kicking around at work?
We’ve been playing around a bit with Berkeley DB Java Edition, specifically the High Availability components (BDB JE HA…for short). If you’re like me, you may not have realized that there was anything more to the Java edition of BDB than the local B-Tree goodness. As it turns out, BDB JE has an HA component, which allows you to quickly set up networked, replicated BDBs. Start a single, master node and then start up replica nodes, letting them know where the master is. The nodes use a Paxos algorithm to determine a leader, which will be handling all writes for the duration of their tenure as master. The master, predictably, replicates to the replicas. Both the replicas and the master can serve reads, meaning that you can scale reads by adding more replicas.
The HA environment manages itself. Nodes come and go, new leaders are elected, events are sent back and forth notifying the new master and replicas of their statuses. Because BDB JE is a library, developers implement the application logic on top of the local database. Applications (a web application, for instance) receive read and write requests. Reads can be fulfilled locally. Writes can be handled locally by the master. If a write request comes to a replica, the replica must consult it’s knowledge of the HA topology and proxy the write request to the application running on the master.
Our question of the day was how quickly BDB JE HA can elect a new master if the current master is taken out of service. Not content to test with a handful of nodes, we burned an EC2 machine image containing a simple BDB JE HA application. The application was simple: set up a local BDB and either wait for other nodes (if it’s the first) or connect to the first node (if it’s not the first). An initial instance of the AMI was started to get the cluster rolling. One the instance was functional, 59 new instances of the AMI were launched. For these 59 instances, we took advantage of EC2 user-data to provide instance configuration information, which included the hostname and port of the original instance for the other 59 instances to connect to. With this in place, the 60 instances automatically configure themselves into a replicated cluster.
Each node logs the result of a leader election. So we log into a single instance, cat the log and determine which node is presently the leader. Logging into that node and killing the Java process causes the remaining 59 nodes to immediately switch from the REPLICA state to the UNKNOWN state. This indicates that a leader election is underway. After a period of time, one of the 59 nodes declares itself the new MASTER while the other 58 nodes return to life as a REPLICA. Below I have included the log output from this process…
db-ip-10-170-31-125 (Wed Feb 16 18:26:59 UTC 2011) - REPLICA
master is db-ip-10-170-85-7
db-ip-10-170-31-125 (Wed Feb 16 18:31:47 UTC 2011) - UNKNOWN
db-ip-10-170-31-125 (Wed Feb 16 18:32:22 UTC 2011) - REPLICA
master is db-ip-10-170-99-156
Here you can see the node starts life as a replica. At 18:31:47 the current master (db-ip-10-170-85-7) is terminated and the cluster goes into the UNKNOWN state. At 18:32:22 (35 seconds later) the node goes back to the REPLICA state and db-ip-10-170-99-156 is crowned the new cluster master. Repeating the process, murdering the new master, yields a much faster second election…
db-ip-10-170-31-125 (Wed Feb 16 18:51:50 UTC 2011) - UNKNOWN
db-ip-10-170-31-125 (Wed Feb 16 18:51:53 UTC 2011) - REPLICA
master is db-ip-10-170-90-48
This time the election takes only 3 seconds. After repeating this cycle another 5 times, the elections hold steady at around 2-3 seconds. Bear in mind, the master DB is not under any write load during this process, so these are ideal conditions. The leader election takes into account which replica is most caught up replicating the master, so a cluster experiencing higher write volume might take longer to elect a new master.
Anyway, this was a quick, fun project. BDB JE is a nice tool to kick around and getting more familiar and comfortable with EC2 is never a bad thing.
Here’s to it not taking another year and a half before my next post.
Migrated blog to WordPress.com
I’ve migrated my blog to WordPress.com. Assuming I’ve done everything correctly, this post will still show up in readers like Google Reader, which should be following the HTTP redirects I’ve recently created. Pray for me and my blog.
Writing a simple Twitter streaming client
I spent a little bit of time tonight and wrote a simple Twitter client, which utilizes the new Twitter streaming API. The new API from Twitter allows you to subscribe to an HTTP stream of public status updates from Twitter users. Using the Apache HttpComponents HttpClient and Json-lib I was able to quickly whip up a Java command line application. The application connects to the “spritzer”, which is more of a drip than a stream when compared to the other streams you can connect with: gardenhose and firehose (both of which require approval from Twitter before you can access them).
Twitter returns each status update on a single line in JSON format (you can also get a carriage-return-delimited XML format), making parsing relatively simple. I wrapped the InputStream I get back from HttpClient in a BufferedReader, allowing me to simply call BufferedReader.readLine() and pass the result to the Json-lib JSON parser.
In the end, my simple client produces the follow output, giving you a sense of what’s possible. I can imagine this API being useful for realtime data mining or visualizations.
Tweet from Shop @ HART market (HARTmarket) Vision Shop \ Fluoroperm 30 Contact Lenses http://cli.gs/JXLuD -------------------------------------------------- Tweet from Hinrich von Donner (hinrichd) @ntvde Kommen bei Euch auch noch "normale" Nachrichten? Langsam nervt's -------------------------------------------------- Tweet from Kelvin Cambridge (supersidekid) If a lifetime is how long you want me for then thats wats you will get http://bit.ly/SrRf7 Maxwell called Lifetime -------------------------------------------------- Tweet from VeolaJBanner (VeolaJBanner) Black pretty lady 20 years old looking forward to hot chat at weekend http://tinyurl.com/pg5h9x - free online adult dating! -------------------------------------------------- Tweet from Sonnie Tease (sonniestack) @admit_one checkers, where, haha. -------------------------------------------------- Tweet from Shanda (ShandaP) Clearly i am on a mission today -------------------------------------------------- Tweet from Krissi Balut (krissibalut) @Connor_Anderson ferris wheel? Where? -------------------------------------------------- Tweet from Justin Kerr-Stevens (jkerrstevens) RT @lovisatalk: RT @govloop: GovLoop parties - DC w/ @corbett3000 http://bit.ly/Cho83 on 7/16; San Fran http://bit.ly/c1jlu #gov20
In any economy, invest in…yourself
I’m sure the two or three people still reading this blog are wondering what I’ve been up to with all of this free time now that I’m not posting here anymore. Well, since December I’ve been investing heavily…in myself.
Really, it began back in June when I went vegan. I’d always considered switching to vegetarian for the health benefits, however Lisa and I had just finished watching the “Animal Rights” episode of 30 Days. After seeing the horrific conditions on factory farms, neither our stomachs nor our consciences would allow us to continue supporting such cruelty, disregard for common decency and a complete lack of regard for public health. Since then, we haven’t eaten any animal products (no meat, eggs, dairy, etc) and we’ve stopped purchasing other animal products, such as leather and wool. As a result, Lisa and I have significantly reduced our global footprints (more than one-third of all fossil fuels produced in the United States are used to raise animals for food). The food tastes great and I don’t miss a thing.
In December, during a long vacation, I decided it was time to work on my strength and conditioning. Not wanting the typical gym experience, I signed up for rock climbing lessons. Thirteen weeks later, I’m still going strong. I’ve been climbing 2-3 days each week, tackling harder and harder stuff. As a result, I dropped seven pounds since the start of the year (222lbs in January to 215lbs in early March) and significantly added to my upper body strength. More recently, I signed up for CrossFit classes in mid-March. CrossFit is an intense, full body workout utilizing aspects of Olympic weight lifting, gymnastics and all the sweat you’ve got. I’m just finishing up the last week of the fundamentals class and I’ve dropped an extra 3lbs over the last 5 weeks while improving strength through all muscle groups as well as my stamina. CrossFit and climbing combined have really woken up the dormant 170lb college kid in me. He wants out, he just has to finish chewing his way through the remaining 40lbs+ of fat.
With CrossFit coming to an end, I’m already looking for the next thing. I have ideas, but I haven’t decided on what to do, yet. More climbing classes. More CrossFit. Possibly pilot lessons. I’m leaning more towards pilot lessons since it’s new and I’m, evidently, all about new things this year. Whatever happens, I just wanted to share this with people. In a period where all of your investments seem to be losing value day after day, investing in yourself never loses value. Keep challenging yourself by learning and experiencing new things, a high rate of return is virtually guaranteed.
Demo Y!OS/YAP Application – "Chatty Kathy"
I gave a talk at work today to the Hack Lunch crew on Y!OS, YAP and the Y!OS PHP SDK. Part of the talk entailed building a sample YAP application using the PHP SDK so I could show people how it’s done.
The application I built, which I call “Chatty Kathy“, grabs all of the Yahoo! Updates generated by your connections and distills it down to a list of your connections (sorted by who generates the most updates) and their applications (sorted by the ones they use most often). It’s not super interesting (although it helped me to find some new applications) and it’s not super complicated (I wrote it in about two hours last night), but it does provide a decent demonstration of how to build an application making use of Yahoo!’s social platform.
For anyone interested, I’ve made the source code available.
Using Ruby to talk to Yahoo! Mail
I’m doing a little work in Ruby this weekend and happened to write a small class for talking to the Yahoo! Mail Web Service. It only deals with the Yahoo! Mail bits, it doesn’t handle Browser Based Authentication (BBAuth). However, I thought I’d share it with everyone to show you how rediculously easy it can be.
To start, you’ll need the net/http (included in Ruby’s core) and json (oddly absent from the core, but available nontheless) modules. The client itself is a mere 18 lines long, although I’ll admit there’s very little error checking being done in this barebones version.
require "net/http"
require "json"
class YMail
def initialize(appid, cookie, wssid)
@appid = appid
@cookie = cookie
@wssid = wssid
end
def method_missing(method, argument)
body = JSON.generate({"method" => method.id2name, "params" => [argument]})
url = "/ws/mail/v1/jsonrpc?appid=#{@appid}&wssid=#{@wssid}"
http = Net::HTTP.new("mail.yahooapis.com")
res = http.post(url, body, {"Content-Type" => "application/json", "Cookie" => @cookie})
JSON.parse(res.body)["result"]
end
end
Using the YMail class is simple enough.
ymail = YMail.new(applicationID, bbauthCookie, bbauthWssid)
puts "Number of Folders = #{ymail.ListFolders({})['numberOfFolders']}"
You’ll have to pass your application ID as well as the BBAuth cookie and WSSID returned during the authentication process to the YMail constructor. However, once you’ve done so you can easily call any method available in the Yahoo! Mail Web Service and get back a Ruby hash to interrogate.
Beginning Rock Climbing
I decided this year that I was going to spend more time focusing on things I enjoy and less time obsessing over things I hate. As part of that, I signed up for the Beginner Rock Climbing Course at Planet Granite. So far, I’ve completed the first two weeks of the course and it’s been pretty fun.
During the first week, we focused on climbing safety. We learned how to get our harnesses on, how to tie in as a climber, how to tie in as a belayer and how to communicate with the climber/belayer during a climb. We also did a little bit of climbing, mostly to let ourselves get accustomed to belaying in preparation fo the belay test we’d all have to take.
During the second week we left the harnesses and the ropes to go hit the boulders. Bouldering is nice because you can climb solo, meaning you don’t need to find a partner to accompany you to the gym to get in a workout. We did some more safety stuff (learning to spot a climber, learning to jump/fall off the wall) and then proceeded to do some climbing. I completely burned out my arms during my first bouldering session, leaving my forearms and hands tired and sore for the next two days.
I’m really enjoying it so far. I was mostly looking for a gym experience that didn’t simply involve picking up some weight, setting it down and repeating. In that regard, rock climbing has been perfect. I can climb several different routes and get in a good workout without it feeling like a workout.
As for Planet Granite, I went to the one in Sunnyvale. It’s a really nice, huge facility that includes the bouldering walls, higher climbing walls, a full gym, locker rooms and yoga classes. It’s a bit more expensive than a normal gym membership ($70/month), but I think it’s worth it given everything that’s available for you (yoga classes are free with membership).
I have two more weeks of class left and when I’m done I’m pretty sure I’ll get a membership and then cancel my Gold’s Gym membership. Climbing has just been much more fun. It feels more like a sport or playing on the playground (the fact that there are always kids there helps it to feel more like a playground).
Give it a try, I think you’ll like it. Here’s some video to give you a taste.
Welcome to the newly hosted unclehulka.com
During my long vacation, one of the tasks I set for myself was to migrate my site off of DreamHost. I’ve been with DreamHost for a while and they’ve mostly provided me with decent service. However, all too frequently I’ve found little things that have bothered me:
- Occasional downtimes while I want to do something. I’m a night person and that’s generally when sysadmins think it’s a good time to take the system down since the users are sleeping. While that’s true in general, it’s not true when your customers are night owls.
- Shell access was often incredibly slow. System loads on the shell servers were often high and so were the latencies, making typing anything on the command line a brutal experience.
- You’re somewhat limited in what you can run. DreamHost picks out the Apache version and whatever modules are installed. They provide a base install of PHP, but (if you’re inclined) you can build your own version and run it using FastCGI. While it’s possible to do this, it’s kind of a pain in the butt.
So I’ve recently switched over to Slicehost. Slicehost does virtual private hosting. You get a virtual machine and are free to run whatever software you want on one of their OS images (they have several Linux flavors to choose from). Anyway, I’ve recently moved all of my unlcehulka.com material over to Slicehost, so…uh…welcome! If you’re interested in trying out Slicehost, feel free to use my email address when signing up to give me the referral (rckenned AT yahoo.com).
I’m hoping to use my newfound hosting freedoms to do some more interesting things with the site. I’m currently working on a simple side project that I hope to launch to the site in the coming weeks. In the meantime, if you see anything broken on the site, feel free to mention it in the comments and I’ll take care of it.
It's Craptastic!
Poop by gtmcknight
Today was my day off. Like any day off, I slept in. When I (finally) awoke, I thought about what I should do today. Among the things that came to mind was to finally go and purchase and set up a new Tivo HD DVR. I know, at this point you’re saying, “but I already know how bad that DVR is, this is nothing new.” If you thought that and left the article, then you missed the punchline. It’s true, the Comcast HD DVR really is the be all end all of shitty DVRs. It crashes, it’s slow, it’s dumb (literally, it doesn’t understand when I say “only record new shows” that I mean I don’t want it to record reruns) and frequently needs a swift kick in the power plug. It’s also goddamn expensive at $16/month on your monthly bill. Tivo’s monthly service costs less than that, so I figured I could save some money (yes, not yet since I have to recoup the $300 for the Tivo hardware first) while having a superior DVR experience.
In order to use the Tivo HD DVR with your Comcast service, you’re going to need a cable card. The cable card handles all of the signal decryption stuff going on in the Comcast supplied DVR. In order to get one, you have to either go into a Comcast office or have them send a technician out to you. Since getting a technician would take days and I’m impatient, I opted to visit a nearby Comcast office to pick one up. That’s where the first problem occurred. I went to the Comcast website and tried to find local offices where I can get equipment. Unfortunately, you can’t find those offices on the Comcast website. You can however find “Payment Centers“. That didn’t sound like what I wanted, still…I decided to check them out. Maybe a local Payment Center doubled as an equipment place. Searching turned up three nearby offices. According to the website, the only equipment available for pickup at the offices was Cable Modems. So…I called 1-800-COMCAST. After faking out the phone system (hit zero twice to immediately go to customer service) I was routed to someone who informed me that the nearby Milpitas office could give me a cable card (even though the web site doesn’t say that they can, evidently Comcast doesn’t want you to know about cable cards). So began my journey.
I drove over to the Milpitas office, which is hidden way in the back of a shopping center. I walked in and a woman at the counter (with a freshly pierced hand, ouch!!!) asked if she could help me. I informed her that I was in the market for a cable card. She took my phone number and looked up the account. She asked if it was under my wife’s name, which it is, and I said yes. She then told me that because the first cable card is only free to people who don’t already have the HD DVR, it would cost an additional $7 (roughly) and that Lisa (my wife) would have to authorize Ryan Kennedy (me) to pick up the cable card for our account. She would have to call 1-800-COMCAST and give them her authorization over the phone. Stunned, I asked the woman if Lisa were to call in the next 15 minutes, could I just turn around and walk back in the office and get my cable card. She told me that I could.
So I walked outside, shaking my head, and tried to call Lisa. Busy. Must be on a conference call. No worries, I had other errands to run including the grocery store and Best Buy (to get the Tivo). I’ll just hit those to kill some time and then try calling Lisa again. So I drove over to the Milpitas Nob Hill, where I discovered just how much one store can completely screw up store layout. Seriously, who puts the peanut butter way over by the milk in the refrigerated section? Half an hour later, I emerged victorious from Nob Hill. I stowed the groceries in the trunk, got in the car and dialed Lisa again. Still busy. Well, off to Best Buy I suppose.
It was on my way to Best Buy that I started having a self-rant. I wondered (out loud) why it is that I can call up on the phone and order pay-per-view on my wife’s cable account but they won’t let me walk into the store and pick up a cable card. That’s when inspiration struck. I thought to myself, “what would Kevin Mitnick do?” When I arrived at Best Buy, I parked and got out my phone and dialed 1-800-COMCAST. Once again, I wielded the double-zero to great effect, immediately putting me in touch with a live person. I informed them that I needed to authorize someone to pick up a cable card for my account. They asked for my account number and I told them I didn’t know it. They asked for my phone number, which I supplied. They asked if the account was in my wife’s name and I responded that it was, so they asked me to verify with the last four digits of her social security number. I gave the operator the numbers and she asked who I’d like to authorize to pick up the cable card. I gave them my name and she told me I was all set.
I ran into Best Buy and purchased the Tivo HD DVR (this is another story entirely), put it in the trunk and headed back to the Comcast office. I walked up to the desk with the same lady I had talked with maybe 45 minutes earlier. “Welcome to Comcast, how can I help you?” Seriously? Did you have THAT many people in here since I was here last? I responded, “I’d like to get a cable card for my Tivo.” She asks for my phone number, which I give her, and she again brings up that the account is in my wife’s name, to which I respond “yes.” She then tells me, “you have a zero balance on your account”. We stare at each other for a minute and I, finally, respond “uh…okay.” She continues to stare blankly at me so I say “what’s the problem?” She tells me “there’s nothing to pay on this account.” At this point I figure she’s fucking with me, so I remind her that I just want to get a cable card. “Oh, I thought you said you want to pay your bill.” In my mind all I can think is, “holy crap…finally something to blog about after a few dry months.”
At this point she goes into the back, procures a cable card and brings it out and verifies that I’ve been authorized to pick up a cable card for my wife’s account. With cable card in hand, I leave the building, shaking my head all the way back to the car.
The moral of the story is (Comcast, you ought to be taking notes by this point), don’t be lame. The lady in the office could have saved me a lot of time if she had simply done exactly what the person on the phone did and ask me to verify the last four digits of Lisa’s social. Instead she completely passed the buck to their 1-800 phone operators, either because she didn’t know that she could ask for my wife’s social or because she didn’t want to be bothered with work on a Friday. Either way, Comcast you look like clowns.
UPDATE #1!!! (November 2, 2008): After activating my cable card, my internet access went down. Unfortunately, Lisa happened to be working from home at the time, forcing her to go into the office on Saturday because she had some pressing work to finish up. I called Comcast and ended up with the most clueless representative I’ve ever had to talk to. She scheduled a truck to come the next day (I’m not sure why they can’t remotely fix a something that they remotely broke). Anyway, the technician showed up the next day and found out that they’d somehow added a second cable modem to my account. As a result, no internet access for Ryan. After an hour of futzing around with their system, they finally got everything working.
UPDATE #2!!! (November 15, 2008): Today I went to return the HD DVR, making the transition to the HD TiVo complete. I took it back to the Comcast office in Milpitas and explained to the guy there why I was returning it, explaining that I was going to be using the HD TiVo from then on. He took it, scanned it, gave me a receipt and told me I was all set. Thinking Comcast got something right for once, I left for home.
When I got home, however, I noticed that all of my cable channels were…black. No Comedy Central. No Cartoon Network. No BBC America. No HD HBO. I went to their online help and started chatting with a support representative. They informed me that my cable card had been deactivated and that I would have to call on the phone to get it reactivated. At this point I was convinced Comcast had put me on a “make sure to completely fuck this guy over when he tries to do anything” list. I called 1-800-COMCAST and, after some fighting the automated answering machine, managed to get to a live person. I explained what had happened and asked if they wouldn’t mind reactivating my cable card WITHOUT taking out my internet access this time.
The technician took my account information and seconds later the channels sprung back to life. She asked me to check my internet access and I verified that it was still functioning properly. We have no idea what happened to deactivate the cable card, however I was just happy to have finally found someone at that company with half a brain.
I don’t get how Comcast stays in business given how much money they must be pouring into support to offset the low quality of their remaining workforce. Then again, people are dumb enough to pay $16/mo just to have their awful HD DVR.
