What is “Hacker Culture?” Notes From the World’s Largest Social Network and the World’s Largest Particle Collider
The Large Hadron Collider (LHC) at CERN is the world’s largest particle collider, operated by a consortium of European governments and host to about 6000 physicists from around the world. Facebook is the world’s largest social network, with over a billion users and about 13,000 full-time employees around the world. On their face, they don’t have much in common at all. Having spent time in both places, though, I actually noticed quite a few similarities. One easy one is that in both places I was surrounded by the smartest people I’ve ever met. Beyond that, both CERN and Facebook are focused on “hacking” unconventional solutions to problems of unprecedented scale. Their cultures stem from confidence, open innovation and decision-making processes that allow good ideas to advance. Facebook and CERN are similar when it comes to culture and open innovation. Differences become clear when we see how decisions are made.
From One “World’s Largest” to Another
We all know Facebook is the world’s largest social network. I’ve been a Visiting Professor at Facebook twice (for 6 months in 2015 and 3 months in 2016), immersed in the Facebook experience (e.g., n00b orientation, “bootcamp” for new software engineers, “data camp” for new data scientists) while still a bit of an outsider because I knew I’d be leaving to go back to my academic life at Northwestern, where I’m a professor in Communication Studies. It’s been incredibly fun and interesting, and reminded me of my experience with another “world’s largest,” the Large Hadron Collider (LHC) at CERN. As a graduate student, I spent 3 months living on-site at CERN to observe physicists at work in their natural habitat. For the uninitiated, the LHC is a 17-mile underground tunnel containing 2 detectors, each roughly the size of an office building, designed and built by ~6000 physicists from around the world to find the Higgs Boson, part of the “standard model” of physics.
Right, so CERN and Facebook are both really big things, but they’re also really different. CERN is bureaucratic (e.g., administrative staff regularly say “it’s impossible” when you want to do something), government-run (e.g., 50's-era linoleum floors to show that every taxpayer penny funds science) and steeped in the traditions of physics (e.g., conference rooms named after great physicists like Curie and Dirac). Facebook is relatively flat and open (e.g., anybody can ask questions at Zuck’s weekly Q&A), has the ethos of a successful tech firm (e.g., generous benefits, abundant snacks and a new building designed by Frank Gehry) and is steeped in a tradition of irreverence to tradition (e.g., conference rooms named for pop culture references, like “Hello. My name is Inigo Montoya. You Killed my Father. Prepare to Die.” in MPK20 Zone 5 ).
These examples highlighted — at least for me — some potentially useful lessons for thinking about what happens when we bring together large numbers of very smart and talented people to address problems on a scale the world has never really grappled with before. Here are some thoughts on that.
Where Does “Hacking” Culture Come From?
In the sense that Steven Levy uses the term (which is importantly different from media portrayals of “hackers” as malicious thieves or antagonists), “hacking” is all about finding clever solutions to hard problems. It’s is a central part of Facebook’s culture and that of Silicon Valley more broadly. Facebook hires programmers with a knack for unconventional solutions, and hacking is drummed into the organizational ethos at every opportunity. ‘Hacker Square’ is where people gather for company-wide events. Construction projects are labeled as ‘facilities hacks’ and room decorations are called ‘space hacks.’ The primary language in the codebase, developed at Facebook (recently open-sourced) is ‘Hack.’ And so on.
Coders at Facebook are often hired because they can hack new solutions to hard problems. At Facebook that means building a scaleable codebase for products that can be reliably delivered to over a billion users; constructing the underlying databases and data models to provide content in real time; and finding sustainable ways to store data — such as photos — for the billion plus people who use Facebook every day. What’s new here aren’t the problems themselves (see, e.g., Friendster, Orkut, MySpace), but the scale. And good solutions need not be conventional, but they need to be clean and make Facebook better.
What’s more surprising is that, once you look past the “It’s impossible”-saying bureaucrats, CERN has a very similar culture. CERN doesn’t call it hacking, but the challenge there is one of building both a human organization and a massive physical apparatus to collide protons and capture the activity of countless resulting particles — plus storing and analyzing the massive amounts of data those particles generate. To be sure, these challenges are different. Compared to data about particles, for example, data about people are more complicated because people have privacy concerns and live longer. As with Facebook, though, the basic problems at CERN aren’t new (see older colliders like SLAC, Tevatron and KEK). What’s new is the scale in terms of size, energy and data. As a result, high-energy physicists take on all sorts of problems unrelated to the core physics problems, such as figuring out how to cool giant magnets place gold filaments in muon drift tubes with micron-scale precision, or write code to analyze massive data sets in parallel. They do these things mostly without regard for convention, with a focus on solutions that work and enable good physics.
All of this is to really say that hacking culture and its ensuing innovations can exist in all sorts of places, not just tech firms like Facebook. Where does it come from? I have a few ideas.
Cultural Confidence
Fundamentally, hacking is about more than writing code. It’s about looking at the world and believing you can come up with a better solution. Everyone at Facebook is encouraged to “hack” — not just engineers. This takes some confidence and there’s strong confidence built into the fields of Computer Science and High Energy Physics.
In Physics, there’s a long history of the field believing itself to be at the top of the science totem pole. This stems from contributions to the mid-20th-century war effort and is reflected in massive funding during the Cold War era for the work that led to the LHC. Physicists have a history of taking on non-physics tasks in doing their work, and they have often been successful. Recall that HTML and the first (text-only) web browser came from physicists at CERN looking for a better way to share their research. Physicists aren’t afraid to act as programmers, engineers or construction workers. They shun outside expertise. For example, I was once on an email discussion among physicists discussing a grant proposal and some of of them were quite explicit about seeing no reason to bring in experts on collaboration, such as my advisor, who were part of the conversation. They were physicists. They had collaborated before. They knew what needed to be known.
The same is true, in some ways, in Computer Science, where the impact of the field has been huge (see e.g., the Internet, Facebook, Google) and engineering new solutions without consulting alleged experts and using conventional tools is routine.
This combination of confidence, scale challenges and ubiquitous building skills is potent. At both CERN and Facebook, people often believe that what’s out there in the world, such as off-the-shelf tools for everyday tasks is inadequate. Often this is true. For example, in Bootcamp at Facebook I worked briefly on internal tools for Facebook — many of which had outgrown off-the-shelf solutions because of the sheer volume of data Facebook has. The company has also developed at least one tool for just about everything (including, but not limited to, data storage/analysis, writing code, scheduling meetings, requesting vacation time, finding and booking rooms, reviewing code, tracking tasks for everything from fixing bugs to distributing food menus, etc.). Physicists at CERN have not built quite as many tools, but they can claim credit for early video conferencing systems for their meetings, a calendar-based meeting coordination system widely used at CERN, and others.
What’s unique here is not that existing tools are inadequate. It’s that people respond to inadequacy not by whining, but instead with the confidence to engineer a new solution. That’s what happens when you hire people who want to build the biggest stuff in the world.
Moving Fast & Breaking Things: How Stuff Gets Done
How does this stuff actually happen in these places? For the physicists and coders on the front lines in both places, there’s actually a lot of similarity.
First, both Facebook and CERN give people substantial freedom to experiment with new ideas. At Facebook this means giving engineers servers on which they can write and test their own features on a fully-functioning (but private) version of Facebook. At Hackathons or even just for fun, for example, anybody with some programming skills can implement and test features. This happened last summer when some interns developed a tool to “pridify” (i.e., add rainbow stripes to) their Facebook profile photos, which they made available internally and then rallied support from employees to roll this out as a real product within just a few weeks. At CERN, freedom means physics groups can work independently on solutions to problems in designing/building detector components or analyzing data. How to build muon drift tubes (a part of the detector), for example, wasn’t dictated by experiment leadership. It came from physics groups.
At both Facebook and CERN freedom also means that access to code and data are fundamentally open. If you’re curious about how something works or if you could change it, you can look. And if you want to make an argument that something is a problem — even if you’re not an engineer — you can typically find the data to back up your thesis. Ultimately, both CERN and Facebook see themselves as meritocratic, in that those who advance the best solutions to problems should ultimately advance. At Facebook, t-shirts and posters trumpet that code and data “win arguments” (implicit is that things like seniority and shouting don’t win them). In organizations this large and decentralized, though, there are a lot of ideas flying around and it’s not always immediately clear that the best of those will surface.
One needs to not only have good ideas, but also make those ideas visible and garner support for them. It’s not enough to build the “pridify” tool; it also needed attention from others to become a product. Quarterly hackathons, day-long coding sessions where anybody can work on any idea, at Facebook can provide a shortcut to high visibility by potentially making ideas visible to company leaders at the review forums that follow each Hackathon.
Overall, being visible means being known for good work (by publishing influential analyses in internal reports at CERN, or internal Facebook notes and data dashboards at Facebook), being selected to give presentations (at experiment meetings or company- and division-wide meetings), and actually having code or results embedded in the product at either CERN or Facebook. The goal in both places is to be known for having impact via clever solutions to hard problems, and to have that impact be noticed and rewarded via inclusion in the larger project or product.
Both confidence and the ability to quickly build, test and access stuff seem to be key components of the hacker culture. How are decisions ultimately made, though?
How Decisions are made
Sometimes good, visible ideas conflict with each other and a single decision needs to be made. Here there are some important differences between CERN and Facebook. At CERN, key decisions meant extensive deliberation in formal meetings and informal discussions — typically over coffee — often involving a senior collaborator separately chatting with those in disagreement, to see if a compromise might happen (one interviewee called this “managing by coffee”). If compromise was unlikely, they convened a peer review panel to hear presentations from each competing perspective, understanding that the panel’s decision was final.
Facebook, unlike CERN, isn’t dependent on voluntary contributions; everybody ultimately reports to Zuck. But it does strive to maintain a flat and open culture. While Hackathons partly show that the culture is open, they also show that decisions about what is included in products ultimately rest with company leaders. The culture is open in that just about everybody has the resources, via code and data, to experiment with and advance their ideas. They won’t win every argument and many ideas won’t advance, but people have the resources to try.
As a researcher, this gets interesting to me when we think about bigger differences between Facebook and CERN. CERN is bureaucratic and it can be very hard to get some things done, but it is fundamentally a research lab. The bureaucracy has control over things like lab facilities (e.g., space, construction protocols, safety), but very little to do with “doing physics” or publishing papers. Decisions about physics and publication will pretty much always be made by other physicists, and not bureaucrats.
This isn’t true at Facebook. There are similar structures (Facebook will bristle at my calling them bureaucracy) that control things like facilities, IT, security and food services. These teams are great and do all they can to support the hacking culture. For example, my team needed a ladder from facilities for a decorating project (sorry, space hack), and were told not “it’s impossible” but “Ok, but you have to go through ladder training first” (upshot: “Be safe. Don’t become a ladder statistic!”).
Being a fast-growing corporation, and not a research lab like CERN, Facebook also has administrative structures for things like public relations, communications and policy. These structures ultimately exist to protect Facebook, but they often do this by constraining and limiting the openness of hacker culture, sometimes with regard to releasing things like papers about research results to the public. In contrast to CERN, these structures don’t reflect the traditions of academic freedom or the openness of publishing. There’s not as clear a line between what’s open to be “hacked” and what will be decided administratively.
Some Lessons
What can we learn from these observations? One key point is that hacking isn’t just about coding. Hacker culture rests on a foundation of confidence, power to innovate and respect for how decisions get made. If any of those things go away, the whole culture may be threatened. Facebook clearly has the first two of these along with a history of making many good decisions and adapting in the face of less good ones. Physics has arguably continued to scale successfully due to strong community devotion to these principles even as experiments grew to include thousands of collaborators. A key challenge as Facebook and other organizations like it grow, though, is preserving a culture where the people doing the hacking (in all forms) believe that their work has the opportunity to advance and have impact.
That means it’s also important to pay attention to the ways that people’s work can become visible. That is to say that hacker culture isn’t just about solving problems with unconventional solutions. There need to be ways for people to notice those solutions. At Facebook this is via Hackathons and openly being able to develop and test new features (like the “pridify” tool). At CERN this was via internal committees and reviews, and being selected to present work both within and outside of the collaboration.
Finally, to keep hacker culture going as things get bigger, watch out for the intersection points between hacker culture and administrative structures. At Facebook an interesting case to consider is the process by which research papers are internally reviewed by legal and communication staff for outside publication. Researchers at Facebook are often excited to share their findings, in the spirit of hacker culture and the openness of academia. Administrators are rightfully concerned about possible harm to Facebook or its reputation if results are misinterpreted or misrepresented by the press. Because nobody really knows what will happen if a paper is published, it’s hard to bring data or code to bear on this argument. It’s also hard to weigh the intangible benefits of new knowledge in society or flattering media portrayals against any possible reputation risk. I’m not suggesting that one side or the other is always right, but that intersection points like this can be sources of tension in this balancing act.
