Welcome to Rebase 38. Suggestions for projects to cover are always welcome, check out the criteria here. In the meantime, check out this preview of some neat visualizations using the GitHub API of how developers are connected:

Featured Project

hiphop is Facebook’s PHP to C++ source code transformer that drops several language features such as eval() in order to statically convert and compile it with g++. This gives Facebook a huge CPU reduction compared to using Apache and the Zend PHP engine, which really adds up when you have 260 billion page views per month and 30,000+ servers. (More numbers here.) The idea behind hiphop is that most high-traffic PHP sites eventually have parts that get rewritten to C/C++ for performance reasons, so this transformation engine allows more engineers to continue writing PHP instead of a lower level language. There’s plenty of information on the wiki on how to get started and run it with nginx. Just keep in mind, hiphop isn’t for everyone, but it could help if you’re running a large PHP app across multiple servers.

Notably New Projects

Oileide is an auto-bookmarking library that can really help out with perma-linking pages that are very AJAX intensive. The gist is this: load in oileide’s JS/CSS scripts, use the unobtrusive helpers to load content, and then the IDs you set on links will be autoloaded if someone reloads the page. From the README, if you have:

<p id="my-content"></p><a href="test.html"
    rel="oileide[my-content]" id="mylink">Click Me</a>

Clicking this link will add #mylink to the location in the
browser, and load the body content of test.html into the 
<p> with id="my-content"

This library has no other dependencies (like jQuery) but the idea could easily be extended or adapted. If you’re doing any AJAX content that needs to be linked, check this project out.

sleeping-wolf is a RubyGem that synchronizes tickets in ticgit, a simple git ticket tracking system that uses your local repository, with GitHub Issues. Sounds great, but it’s super-alpha and needs help from the community. Better documentation for ticgit would be great to have, perhaps a cute logo, and the actual code for synchronization (perhaps using the Issues API?). This project could really help out GitHub users that like to keep local backups of their project issues, so get hacking!

funfix is a Python fixtures library that can whip up sample test data using Google App Engine’s Datastore API. Heavily inspired by Rails’ fixtures and factory_girl, funfix includes plenty of examples on its README about integrating it with your app, and some neat features including deriving fixtures from other fixtures. This project was extracted from CrashKit, which may be worth a look for those into ‘Exception-driven development’.

squawk is a command line tool for parsing and running SQL-like queries on your log files, which is also written in Python. I know, that sounds scary, but it’s really useful if you’re not already a Unix wizard. You can run queries such as SELECT COUNT(1) AS n, remote_addr FROM file WHERE status = 200 on Apache or nginx log files, or even GROUP BY status to get an aggregate count of all statuses your web server has dealt out. With subqueries and aggregates, this tool could definitely save you time when hunting in log files. Combined with clarity for a web interface view to grep and tail -f, this is pretty much a sysadmin’s dream.

This is just great: NCSA Mosaic on GitHub at http://github.com/alandipert/ncsa-mosaic.

This now joins the Quake source as one of my favorite old school projects.

(Hat tip flangy.)

It’s about time – the GitHub drinkup is moving to the peninsula! One week only, don’t miss out! If you’re a peninsula dweller like me, join me at CityPub in Redwood City at 8pm next Thursday, March 11. It’s right by the CalTrain stop, so you have no excuse:

2620 Broadway Street
Redwood City, CA 94063-1533

Hope to see you there!

Picking up where Kyle left off in his Branch List post, we're all very excited to announce a new feature designed to ease the process of comparing two points in a repository's history. It's called GitHub Compare View and it's going to change the way you review code.

The Compare View brings all information needed to determine what changed over a series of commits onto a single page: a condensed commit list in chronological order, followed by a rollup diff of all changes between the two points, followed by any relevant commit comments. All in the same place and with a single well-defined URL.

It's a versatile feature with many potential uses. Some of our favorites are detailed below with links to live examples of Compare View in action.

Reviewing topic branches before merging

Reviewing topic branches is a fundamental activity for anyone maintaining an active open source project or working with a team on a private project. Until now, the easiest way to do a thorough review of a complex topic branch was to drop down to the shell and run some combination of the git log, git cherry, and git diff commands. With Compare View, we've taken that process and put it behind a single URL, so jumping into the review process is usually as simple as following a link.

Follow the examples below to see it live:

• jQuery require support branch (pictured above)
• Mustache pragma support branch
• Homebrew Pathname.rename branch

Generating comprehensive change logs for releases

While Compare View was initially designed to supplement our code review process, we quickly found that it was useful in a variety of other scenarios. Projects that are diligent about tagging can use Compare View to generate a comprehensive list of changes between any two releases.

A few examples in the wild:

• Changes between the Sinatra 0.9.4 and 1.0.a releases (pictured above)
• Changes between the Rails 3.0 beta and current master
• Changes slated for the next version of Git

How to customize the commit range

The commit range determines the starting and ending point to use in the comparison. It looks like this:

Here, we're reviewing the proxy_owner topic branch using the master branch as the starting point. Hitting the button switches the starting and ending points. Clicking the starting or ending point label brings up the ref selector:

Enter any branch, tag, or commit SHA1 in the little box there and get an instant preview of the newly selected revision. Hit Save & Refresh once you're happy with the selection.

Getting There

We've seen what the Compare View feels like and how to change the commit range once there, but how do you get to a Compare View in the first place?

• The Branch List page. Click the Compare button next to any branch to jump into a Compare View with that branch as the ending point. See Kyle's post introducing the Branch List page for more information.

• Push and Branch Create events. All push events with more than one commit now link to a Compare View over all commits included in the push. All create branch events now link to a Compare View between the repository's default branch (typically master) and the branch head. This effects dashboards, repository timelines, and activity feeds.

• Service Hooks. The IRC and Campfire service hooks now drop a Compare View link when more than one commit is included in the push. More service hooks will follow.

Compare View URLs

We wanted Compare View to be something we could link to from external sites and services anytime we were referring to a range of commits in a git repository. As such, we thought it would be worthwhile to document the basic structure of a Compare View URL:

http://github.com/<USER>/<REPO>/compare/[<START>...]<END>

Where <USER> and <REPO> are obvious, and <START> and <END> are branch names, tag names, or commit SHA1s specifying the range of history to compare. If <START> is omitted, the repository's default branch is assumed.

Big Plans

Compare View is the first of many code review related features we plan to introduce this year. We'll be incorporating Compare View into other areas of the site and developing entirely new features with Compare View as a core component.

That being said, we felt it was important that the basic task of comparing two points in a repository's history stand on its own and have a well defined URL so that the feature is useful in a broad number of circumstances. If everything goes to plan, you'll be seeing Compare View links tossed around on mailing lists, forums, IRC channels, Campfire, blog posts, release announcements, issue trackers, etc. in the very near future.

Git's branching model is one of it's best features. Branches are cheap, fast and extremely flexible. They're great for developing features, maintaining old releases, or just plain experimentation.

If you spend a lot of time with git, you'll also find that there's a lot of really useful information to be discovered in the way git can compare various branches. We're using some of this information to generate our new branch list page (you can get to this page from the 'Branch List' subnav item under Source).

Check in on your topic branches in one glance

Not only do these new branch list pages show you which branches exist on your remote, but you can see at a glance how they compare to any branch.

Each branch has what we call a divergence graph. On the left side of the black bar we show how many commits that branch is behind (commits in master not found in the branch). On the right side, we show how many commits that branch is ahead (commits found in the branch, but not in master). The colors of the bars indicate how recent the last commit was.

In that one graphic, you get an idea of when the last time each branch was updated with master, how far along that branch is, and if people have been working on it recently.

These graphs also have some implicit rules that can help when merging:

• No left side bar - This branch is safe to merge with your base branch and there will not be any conflicts.
• No right side bar - This branch has already been merged into your base branch, so it's probably ready to be deleted.
• Large left and right side bars - This branch is probably going to be difficult to merge. There's a lot of unique commits in the branch and in your base branch so the likelihood of merge conflicts is much higher.

Having fun with Rails releases

This view can also be fun to glean some information out of Rails releases. Rails keeps a branch for each point release. If we take a look at the branches with 2.2 as the base, we get a pretty interesting page

Using the divergence graphs, we can see how each of the point releases of Rails compare to the 2.2 release. You can also see that there was almost as many commits from 1.2->2.2 as there has been from 2.2->master (Rails 3 beta).

Compare View

The last piece of the branch lists page is the compare button on each branch. This is an awesome feature--but I think I'll leave it to Ryan to explain in more detail.

According to git-checkout: “You can make changes and create a new commit on top of a detached HEAD”. Git is your Friend, not a Foe explains how this is possible.

Featured Project

mediacore is a audio and video content management system built on top of Python’s TurboGears and MooTools. If you’re looking for a YouTube inspired site complete with comments, podcast publication for iTunes, and a great theme out of the box, look no further. It can handle video or audio posted on other sites such as Google Video or Vimeo, or you can distribute media straight from the site. Check out the demo or even some existing installs of the system in use to see what can be done with it. The project’s site and documentation can help you get started with your own MediaCore site.

Notably New Projects

WebGLU makes developing WebGL applications fun. It provides a series of higher level functions that makes common rendering, animation, and shading techniques easier while still allowing developers to dip down into lower level APIs if necessary. There’s plenty of examples in the repo, and you can get started in only 25 lines of JavaScript. The only catch is that you’ll need the 3.7 Alpha of Firefox to try it out, and then enable it in about:config. One step closer to no more Flash!

serenity is a Ruby library that helps with creating OpenOffice documents (.odt), complete with Firefly inspired examples. You can essentially think of it as ERB, just instead of text or HTML, you’re producing an actual word processing document. The neat part is that you can use Ruby blocks and code inside of the .odt template. Clone away and browse the showcase.

grong is a municipality in Nord-Trøndelag county, Norway and a Gross and ROugh Nameserver written in Go. GRONG can be used as an authoritative name server like nsd, and it comes with a few test built-in responders including an in-progress AS112 implementation. It’s in a very experimental state, but it’s definitely a great example of what’s possible with Go, since it uses Goroutines and also binary protocol reading and writing.

webgac is a dependency manager much in the style of Maven, Ivy, and maybe even a little bit of RubyGems for .NET. This project uses WebDAV to store the dependencies and can be served up by Apache. From the developer perspective, you can use the project’s VisualStudio Addin to configure it and set a custom Import MSBuild target that will pull in the binaries you need. For more info check out the README and this blog post from the author about how it works.

Over the past few months I've been working on a major new version of Ernie, the RPC server I wrote to power GitHub's sharded file server architecture. As a reminder, Ernie is an Erlang/Ruby hybrid BERT-RPC server (packaged as a Rubygem) that let's you expose Ruby modules as an RPC service. It spawns, manages, and load balances between a set of Ruby processes that allow access to the Git repositories.

Over the past four months Ernie has proven to be extremely stable and reliable. Each of our five file servers is handling an average of 50 req/sec (4.3 million req/day), bursting up to 200 req/sec, and transferring over 11GB of data per day (just to the web frontend and jobs; this number does not reflect pushes/pulls/clones/etc.).

Last week I released Ernie 2.0 and upgraded all of our file servers to use it. Last night I released and upgraded everything to 2.1. Here's a breakdown of what's new in Ernie 2.0/2.1 and how we're using these new features to give you an even better GitHub experience.

Native Modules

The biggest new feature in Ernie 2.0 is the ability to define handlers in pure Erlang (instead of just Ruby). These are known as native handlers. Native handlers execute within the Erlang server's VM and therefor do not have concurrency limits like the Ruby handlers. In addition, the roundtrip to an external process is eliminated, boosting overall performance for those functions. About half of the RPC calls that are issued to Ernie are very simple file existence checks. By implementing these actions in pure Erlang, we've reduced the amount of work being done by the Ruby processes and freed them up for other tasks, all of which means our maximum concurrency has increased significantly.

Language Agnosticism

Ernie no longer assumes that your handlers are written in Ruby. You can now use any language to implement your handlers as long as they speak the external handler protocol. Non-Erlang handlers are called external handlers. Currently only Ruby support is included in the distribution but I'll be adding other languages as the need arises or as contributions come in.

Shadowing

As time goes by, we will be converting more and more Ruby to Erlang to take advantage of the native handlers that Ernie 2.x supports. To make this task as simple as possible, Ernie supports a concept called shadowing. If you define an external handler and a native handler of the same name, Ernie will check the native handler for an exported function of the requested name and use it if it exists. If it does not, it will fall back to the external handler. This feature makes it incredibly simple to migrate functions one at a time to pure Erlang without having to change a single line of client code!

Predicate Shadowing

In addition to basic shadowing, you can choose whether to run the native or external version of a specific function based on the arguments. This is called predicate shadowing and is accomplished by returning a boolean from a complementary native function named myfun_pred where myfun is the name of your function. We use this to selectively implement parts of the proxied Grit in pure Erlang.

Connection Priority Queue

Requests can now be classified as either high or low priority. Ernie will immediately process any connections marked as high priority. Low priority connections will only be processed if there are no high priority connections pending. We will be using this feature to keep low priority jobs from saturating the file servers with requests that are not time critical, thereby keeping the servers responsive to website requests. While this specific treatment of the high/low queue is rather rudimentary, I plan to include more advanced strategies in a later release. This is really just the groundwork.

Access Logging

Ernie 2.1 introduces a proper access log to make it simple to track what your Ernies are up to. The log file contains the message type (access or error), the time of the initial connection, the number of seconds between connection and when the request is selected for processing, the number of seconds the request took to process, the lengths of the high and low priority queues, the type of handler (native or external), the priority of the request (high or low), and the first 150 bytes of the request.

The Future

BERT and BERT-RPC along with our Ruby and Erlang client/server implementations have made it possible for us to build a high performance, sharded file system architecture for a vanishingly small amount of money. We currently have five terabytes of active storage exposed via BERT-RPC and are adding a new file server pair every few months. In the long run, I intend to make Ernie the most robust and flexible RPC server available while preserving the simplicity of writing handler code in the language of your choice. Keep an eye on the project, there are plenty more improvements to come!

CoffeeScript (.coffee)

Objective-J (.j)

Haml (.haml)

Sass (.sass)

You may think that the location of this weeks drinkup is a subtle way to cast in our lot with a particular side of the age old conundrum, ‘Pirates or Ninjas?’ It is not. We are an Equal Opportunity Employer and as such we consider applicants for all positions without regard to race, color, religion, creed, gender, national origin, age, disability, marital or veteran status, or any other legally protected status including but not limited to pirate/ninja affiliation. Also, as it turns out, there are no ninja bars. Ninjas if you would kindly rectify this we’ll happily meet on your turf next time.

The Facts:

Il Pirata 2007 16th Street San Francisco, CA 94103 February 25th 9pm

UPDATE: These maintenance windows are now complete.

Starting tonight we will be performing a series of partition resizes on our file servers that will allow us to streamline our backup process. Each file server is broken up into 16 partitions and we will be doing the resizes on one partition at a time. Each resize is estimated to take about 20 minutes to complete during which time that partition and all repositories stored there will not be available. During the resize of your partition you will see a notice on the website informing you of the temporary unavailability and you will not be able to push, pull, or clone your repos via Git.

Thanks for your patience as we improve this portion of our infrastructure.

Ever since we introduced the file upload section for repos, the two requests we’ve received the most have been for a no-flash alternative and download stats. Today we’re introducing both.

Flash-less uploads

We’ve had our reasons for using flash, mainly that it allows you to upload directly to S3 (instead of our server acting as a proxy), and flash can provide a status bar. Today we added a second means of uploading, basic HTML.

Flash will still be the default, but for users that block flash or don’t even have it installed, there’s now an alternative. As a bonus, if you use Chrome it will give you an upload status:

Download stats

The other major improvement we added were download stats:

Note that these download counts are from today forward. We were unable to retroactively calculate downloads.

Today we’re opening up a new site — http://status.github.com. We’ll be using this site (in addition to our twitter account) to post status updates (maintenance warnings, outages, etc). This site is hosted over at Heroku so it will be shielded from any network problems GitHub may suffer.

We take our uptime and availability very seriously (remember, we use GitHub to develop GitHub!) and want to be as open as we can with service disruptions.

Check it out. We’re doing a repeat venue. STOP CRYING. We’ll see you at Thieves Tavern tomorrow, Thursday February 11th at 8:30pm. Here’s a second chance for you to not confuse Thieves Tavern with Whiskey Thieves or Dirty Thieves or Ali Baba and the Forty Thieves or Thieves Thieves Thieves and Thieves like you might have last time.

As always, if you have neat projects you want to show off send me a message! I usually try to keep a balance of languages/domains between the posts, so don’t lose hope if your project isn’t in the latest issue. Just please have a README so you can show others (and me!) how to setup/use your project.

Featured Project

coffee-script is what HAML and SASS are to HTML and CSS, just for JavaScript. This is a fantastic DSL that compiles down to executable JS and adds plenty of awesome features along the way. CoffeeScript offers a terse syntax inspired by Ruby and Python, list comprehensions, existential operators to remove pesky typeof checks, and a reliable inheritance model to boot. The list goes on and on, and for executable examples along with their compiled-to-JavaScript equivalents, check out the docs. Some projects are already cropping up, including rack-coffee and even a rewrite of underscore.js. GitHub’s languages section already recognizes CoffeeScript, so get hacking!

Notably New Projects

nv, or Notational Velocity, is an OSX desktop app that does just that: note taking. It approaches this simple task in a mouseless, modeless way, and user interaction is driven through incremental search. If it can’t find a note with you’re starting to type, and a new one is instantly created and saved. The data’s also not trapped on your own machine, you can hook it up to Simplenote, or to anywhere via WriteRoom or Dropbox. Download it here.

jquery.fortune is a replacement for Unix’s fortune in jQuery. Set up your fortune cookies with JSON, hook it to an HTML element, and you’re all set. There’s a live demo if you want to see it in action. On the more practical side, this could be a great example of how easy it is to make a jQuery plugin. I’m just waiting for the first person to combine this with the jQuery Konami Code.

hubroid is an Android app for keeping track of what’s happening on GitHub while you’re not hooked to your computer. So far, the app can look up the repos you’re watching and your own, forked repos, and the latest commits from a given repo. Being an Droid owner myself, I’m tempted to break out some Java to help contribute. It would be great to view commit diffs or your timeline in the app, or even post gists.

Following three months of near 100% uptime, we’ve just been through three major outages in as many days. I wanted to take some time to detail the problems and what we intend to do to prevent similar downtime in the future.

Outage #1 (02/02/2010 9:55:09AM PST) was initiated by a load spike on one of our file servers (fs1a). When a file server stops responding to heartbeat, the slave server in the pair kills the master and takes over. In this case, the master was not killed quickly enough and the storage partitions did not migrate cleanly to the slave. Cleanup on the split-blain file server pair was delayed due to some inefficient DRBD configuration that we’ve been meaning to update. By rolling out improvements to the DRBD configuration, this type of problem should be prevented from happening in the future.

Outage #2 (02/03/2010 6:10:08PM PST) looked like a power outage at first, since so many machines were affected, but the root cause was the deployment of a faulty DRBD configuration update that propagated to all machines (courtesy of Puppet) and started causing pairs of machines to halt replication to prevent corruption caused by an invalid configuration file. Eventually the load balancer pair was affected and we could no longer even serve the Angry Unicorn page. The way that the servers went down, the number of servers that went down, and the length of time it takes to resync downed pairs resulted in a lengthy outage. There are several steps to preventing this kind of outage in the future. First and most obvious is to maintain tighter control and testing of proposed system-wide configuration changes. We also plan to deploy (well-tested) changes to the DRBD configuration that will reduce cleanup times and automate the startup process for downed machines. These changes will result in shorter recovery times in the event of single failovers and wider machine-level restarts.

Outage #3 (02/04/2010 2:37:08AM PST) was caused by massive load spikes across all five file servers. To prevent extended downtime we marked all file servers as offline (preventing them from going into failover) and looking for the cause of the load. After inspecting the HTTP logs, we identified a Yahoo! spider that was making thousands of requests but never waiting for responses. After banning the spider, the load returned to normal and we were able to bring the file servers back online. We are looking at our rate limiting strategy and will be making improvements over time to get the best performance for legitimate users and the best protection from anomalous behavior.

In order to execute the improvements to various infrastructure elements, we will be having scheduled maintenance windows at 10PM PST over the next week. Most of these changes will not require any downtime, but some of them may result in temporary unavailability of file server partitions. As we perform the maintenance, we’ll keep you updated via the GitHub Twitter account, so make sure to check there for the latest maintenance news.

We sincerely apologize for the recent problems and are working very hard to address each flaw. Stability is one of our biggest goals this year, and I look forward to making your GitHub experience as flawless as possible.