Collaboration and Notqmail with Amitai Schleier
Episode 35
On this episode of Programming Leadership, Marcus and his guest, Amitai Schleier, discuss a new project Amitai is working on regarding reviving an old, but useful, open-source program called qmail. Strategy and collaboration on this project as well as how to manage a project of this nature are discussed.
Show Notes
- The ‘old’ project is called notqmail. @1:10
- Last stable release was in 1998 then it was abandoned. @2:45
- Elders decided to make some changes in 2007 and called it notqmail. @3:49
- Amitai decided to revive this old C code. @7:16
- He wanted to join together the other people still running with qmail or netqmail and collaborate to make the best modern version possible. @10:24
- The best advice was to take everyone’s add-ons and then his own and let the users decide which to implement to avoid egos. @14:04
- Collaboration depends on the properties of the code being worked on. @16:51
- Amitai put together individual persuasive invitations to get people to join his team. @24:19
Links
Transcript
Announcer: Welcome to the Programming Leadership podcast where we help great coders become skilled leaders and build happy high performing software teams.
Marcus: Alright. Welcome to the show, I am super excited to have my close personal friend Amitai Schleier with us. Amitai is an independent software development coach based out of New York and we met recently and have just become the best of friends. Hello, Amitai.
Amitai: Hi Marcus, it’s nice to see you for the eighth time today.
Marcus: I know, right?
Amitai: Just so familiar.
Marcus: Great to see you too. But you know I hear that you were covered by an O’Reilly newsletter who sent out a link to their billion people about a super fun project. I think it’s qmail, is that right?
Amitai: That’s right. So O’Reilly apparently has a newsletter called Four Quick Links, which I am now aware of because somebody brought to my attention that I had a project featured in it. It is derived from a 20 year old piece of software to run an email server called qmail, and the project that I started is called notqmail.
Marcus: Wonderful. Well, here’s the reason I think that if you’re listening to this show, which is by the way not the mail server show, this is the Programming Leadership podcast. Yeah, I know. But Amitai had to ask himself, here is this legacy code, he’s got this private fort and we’re going to hear about it. But when he started telling me, he said, “And I had to figure out how to get other people on board with doing something that it didn’t even necessarily have to be done, the world wouldn’t end.” Right? But you then used your technical leadership skills to wrangle some folks together and things are now moving. So, let’s back up and tell us what is qmail?
Amitai: qmail is some software for Unix systems that was last updated 20 years ago for running an email server. So you all use email, we all use email, sometimes we wish we didn’t, but we do. That means that all of us have email servers. It’s just that most of us aren’t deeply aware of them, we use someone else’s email servers. Our company’s email servers, Google’s email servers, a few of us know how to run our own, and have decided it’s important to us. It used to be easier than it is now. It’s gotten harder.
Marcus: Amen, yeah.
Amitai: Yeah, it’s gotten more and more complex and that’s part of why I have a dog in this fight. But so qmail is software for if you want to run your own email server, you could use this.
Marcus: Now, this code, at least Wikipedia, if you Google for qmail, you’ll find Wikipedia says the last stable release was 1998, is that right?
Amitai: That is right. There is a looser sense in which there was a slightly more recent release, but the last release by the original author was in 1998.
Marcus: Now, how did this code, which is still running around … Well, first of all, have you been using it for a long time?
Amitai: I’ve been using it almost continuously since 1999.
Marcus: Okay. So you have a deep affection and a lot of experience with this code and so here it gets abandoned. Right? Daniel J. Bernstein, the nice Wikipedia article says was the gentleman who wrote this code, and he was just at some point abandoned it, it says. But the license is something called public domain. So what made you decide that it needed to be revived? Does it have a bunch of bugs?
Amitai: Well, so the loose sense in which there has been a more recent release is that up until about 2007 there were some, you could call them, elders of the qmail community who decided that in the absence of an official update, there are a few small changes that everybody running qmail should have, that we believe if the author produced a new version he would include, and because were elders and are respected, this release was considered authoritative in the absence of anything from the one authority.
Amitai: So there’s a small history of trying to update the software. Unfortunately, notqmail authors, notqmail is what they called their patch set, were limited by conservatism, which is to the good in a project like this because the reason that the original qmail was valued so highly was that it was written by an unparalleled security genius. I don’t like to use the word genius, but he really is one of a kind, and even people that don’t care for his personality agree with that.
Marcus: This is Mr. Bernstein.
Amitai: That’s Professor Bernstein.
Marcus: Who also has … Oh sorry. What was that? Professor-
Amitai: Professor Bernstein.
Marcus: Who also has a nice Wikipedia article page and appears to have worked at a variety of places and looks like is a valid genius, but why did he not just keep working with the people who are already so invested in it.
Amitai: He did not have a very strong history of being collaborative on this project, which is sort of a defining characteristic of how the code came to be what it is. He did take improvements from certain contributors, but it was always his choice what went in. There wasn’t any kind of a public repository like we have now with Git and GitHub. Whatever changes he said would go, went and everything else didn’t, and it was his project. So..
Marcus: I think I have worked with people like this. Maybe they weren’t geniuses but they had some of those other attributes.
Amitai: Well, and that was why it was a worthwhile trade off because compared to what we going to run qmail compared to Sendmail, which was, if you’re old enough to remember, a nightmare. This was a coherent, well-designed, really Unix Lino separate programs doing separate things as well that you compose to do what you want. It was designed by someone who had a lot of insight into how this fits together in a coherent way. I didn’t really understand internet email in a deep way until qmail existed. Reading it and understanding it taught me how email itself work. It was that good at design that it explained the concepts to me.
Marcus: Wow. So you’ve got this genius who works primarily alone is my guess, who writes a lot of the code, who is responsible and becomes the, maybe not so benevolent dictator, but the one who decides what’s in and what’s out, and at some point it seems like he just decides he’s not interested in it or he decides he’s going to pursue something else. So it gets left behind.
Amitai: It was always a side project for him, is my impression. He’s a professor of math and computer science. He’s one of the world’s leading cryptographers and that’s what he’s busy with these days. I think the purpose of him getting involved in writing an email server is that he was expected to run some mailing lists and he didn’t want to keep getting himself broken into running Sendmail. So he needed something that he could trust so that he could walk slowly away from it. So as soon as he could walk slowly away from it, it looks like he did.
Marcus: So as you thought about reviving this really old piece of code, and I’m assuming this is old C code.
Amitai: It is.
Marcus: Okay, yeah. So you’ve got all this old C code. I mean, this is legacy code, if anybody out there is listening and you think, Oh we’ve got legacy code, you may not have code from the early nineties running around, but that’s what we’re talking about here. Amitai, how did you go about even thinking about reviving this and was there already a collaborative group or like how did you wrangle people into helping?
Amitai: Well, you know it’s probably true 10 years ago that I wished that someone would pick up the development, but 10 years ago I didn’t feel qualified to do something about it. I didn’t know C, I didn’t understand in enough detail how to organize around some new design that would get people involved. I didn’t have time or energy to spare, and I have maybe a little bit now foreshadowing, but so in the interim of … since I’ve been running qmail, one of the things that really challenges me cognitively is this tension between, it’s the Unixiest design you ever saw.
Amitai: Maybe even more so than the Unix systems that are runs on, which for example like say that you agree that it’s a good idea for programs that need to run in the background for a long time, demons we call them, not to have to independently reinvent how to be demons, but they can just be regular programs that read standard input, write standard output, log to standard error. Then all we need is one other set of programs that can turn those programs into demons. You could call them demon tools. This is exactly what Daniel Bernstein did, and it’s a beautiful design and arguably it’s what Unix should’ve done to make demons possible, but it didn’t. So this is this tension where like Solaris has a boot process and it is not demon tools.
Amitai: So how am I going to start qmail at boot time? And you just have a bunch of these rough edges of decisions that each Cis admin has to decide to make. So I didn’t have a coherent picture of how to resolve all those rough edges but I was deeply imprinted at a young nerd age with them and just set about for the next 20 years slowly resolving these tensions in the guise of package management. So one of my other side projects is packaged sources across platform Unix package manager, and qmail is notoriously difficult to package for a variety of reasons including what we just talked about.
Amitai: And so, piece by piece, increment by increment, I would hammer it into slightly less complicated to manage shape, slightly more automated package shape. And over the last couple of years I started working on my own code because the next thing that I had to do to untangle was add-ons that would make the package simpler. But I had to write my own code to do that. So then I started learning C and I started learning the idiosyncratic brand of C with the Bernstein’s own libraries because he avoided standard I/O as much as possible. You can see the whole picture. You can imagine what kind of a character this is.
Marcus: I’m starting to get the sense.
Amitai: But I got into the game, I got into the game myself, not exactly a fork but not morally very different from a fork. Then a couple, three, four weeks ago I started thinking, it’s really a shame that there are six or seven people I know of who are publicly maintaining their own forks, each of them individually. Everybody has gone in their own direction, I’ve gone in my own direction. I like to think that I pick a direction that’s generically useful, and that if there were qmail, the changes would be accepted, but we can’t know that.
Amitai: And anyway, what I really wish is that somehow we could collaborate and take the best ideas that have been formed among those of us who are all working on this stuff and care about this stuff and join them and provide, you could say like a real success criteria and it would be, if all the people who are still running qmail or netqmail saw what we made as a straightforward, easy, safe upgrade and they upgraded to it. Then carefully we could modernize toward the features that everybody currently needs to manage their own crazy patchtree about. If we could do that, that’d be pretty sweet.
Marcus: So you’ve got that idea of what’s the goal? I mean, that was a really clear description of success and there are, I think you mentioned seven other people floating around the Internets who are maybe as excited about this as you and each of them has been maintaining their own private copy for 20 years. How did you go about getting folks together or what organizing … yeah, let’s start with that. How did you even approach these people?
Amitai: So first of all, there may be more than seven. There are about seven that I’m aware of being active. There may be others who are active and don’t realize there’s anybody out there worth sharing it with, and they may come out of the woodwork when we have our first release. I hope they will, but the first step was I’ve been keeping an eye on this community. I have a list of names of people who I like what they’ve done and I’ve stolen some of their code, stolen, it’s open source, that’s what we do. We borrow it from each other.
Marcus: Right.
Amitai: Adapt it to our needs, but they’ve done stuff that I have valued and benefited from, and just sent them an invitation. But I didn’t do that first. First I got some advice and the advice, the key piece of advice which I got from Llewellyn Falco, he’s a fellow technical agile coach and he has an open source project that gets lots of participation. He was saying, “So if I hear you right, qmail already has a lot of nice scenes and interfaces in the system and that’s part of why you love it so much. Is that right?” Yeah, that’s right.
Amitai: “So it’s probably true that without too much more work and without too much more risk, you could add a couple more interfaces and scenes such that whatever modernization is required could be done as plugins, add-ons, extensions, so that you wouldn’t have to play with the core code. Is that probably true?” Yeah, that sounds like it could be true. “Okay, good. Then the next problem you have to think about is egos, because each of you has your own choices that you’ve been building on top of your own previous choices for years and years and years, and not all of these ideas are going to win. Who’s going to decide?” And this is where he gave me the best piece of advice.
Marcus: I can’t wait to hear.
Amitai: His advice was, “First of all Amitai, you have maybe a special knack for being able to get a project like this off the ground. You have some charisma, you have some persuasion, you’re good with the words, and you yourself are pretty egoless about how code should get shipped, that could make this work. So whatever you do, offer yourself to help other programmers get their code converted to extensions first, and then after that seems like it’s going well, you could add yours. The decision about which implementation of let’s say a SMTP authentication, which is required now but standardized after 1998, so which implementation are we going to take as the good one? That’s not your call, it doesn’t need to be your call. Take as many implementations as people want to give as extensions and let users decide, and it doesn’t necessarily have to be that one wins, it could be 60% of the aspects of this one are the best and take 40% from the other and merge them and then that’s the winner. But you can figure that out later.”
Marcus: That is wonderful advice and I feel like these themes, this is why I had you on the show because this is the kind of thing that software managers and technical leaders have to deal with every day. You’re sitting in a room with smart people, everybody’s got an idea about how it happens. You’ve got an idea because you got promoted because you are a great coder and everyone got egos, and so that idea is just so powerful that other people are going to go first, that other people you’re going to act in a support mechanism for other people not having to prove that you are particularly grand at coding or prove something else, but the thing you’re trying to prove then maybe, and I’m trying to come up with the right words, but the thing you’re trying to prove is that you are there to help.
Amitai: Yeah. That it’s going to be worth it to them to throw in with us, we’re defining a new us. Speaking this out loud to you, I had my own insight just now, pretty often in a team that is struggling. The thing that’s holding them back is some human factor about the team behavior such as ego. Another thing that comes up that I see very often and my bias is clear, I’m a technical agile coach, but even when all the team dynamics are beautiful, the next bottom line that can be the state of the code that can limit what kind of collaboration is possible. I’m noticing now Llewellyn identified it and now I’m hearing it in a new way. This kind of collaboration is possible because of the properties of the code that we’re collaborating on.
Marcus: Go deeper on that. I want to really understand.
Amitai: Yeah, so if we were working on … you could compare and contrast with a monolith and microservices. It’s a good direct comparison and not a 20 year old comparison. So yeah, with a monolith, there’s one piece of code. It may be that it’s beautifully architected, in which case maybe you can decompose things and work in parallel and solve problems separately and whatever it is, however you want to collaborate, if that’s the way you want to collaborate. Or it could be that it’s a more typical monolith and everything’s intertwingled, and there’s always merge conflicts between your story and my story.
Amitai: Versus microservices, at least the promise of microservices is that those dependencies are made clearer and more separate, and so you can work on your thing and if you version it right, I can depend on it either the old or the new. If we avoid cycles, we’re in good shape and you know we won’t have these kinds of conflicts. We can work as parallel as we care to. That’s an example, and I think it’s a directly comparable example of the properties of the code limiting or enabling a style of collaboration.
Marcus: And so yeah I agree. I don’t know that I ever … and this is where I haven’t worked with many software architectures in about five years. So I know microservices is very popular, but do you think most people go to microservices because they want better collaboration? I don’t think I’ve heard that as the primary reason, but I’m really intrigued now is that the driver?
Amitai: Not usually in those words. Usually it’s about deploying independently or risk of going to production or things like that. But those are because those are the limiting factors in the collaboration that they’re having. So nobody usually says it in those terms but ultimately it is those terms.
Marcus: So does your group, the group whoever you’ve put together, whoever has agreed to join the choir or whatnot, do you have an organizing principle or is it just everybody gets a vote as a democracy? Yeah. What the organizing and decision making processes?
Amitai: It’s a little bit fraught because on the one hand I am the person that invited everybody and got this thing started and on the other hand I would love it if a year from now I could back slowly away and nobody could figure out how this project even started. That would be a beautiful outcome. I just get to use what they’re building and I like how they’re building it and I get to update to it and I can find a different hobby, that would be beautiful.
Marcus: Why do you want that?
Amitai: Because I often have to spend a lot of time justifying this hobby to a lot of people. It’s a strange one. There’s other old software I’d like to revive for that matter. So if I can get other people doing this one for me, then I can go to another one.
Marcus: So you want this to exist, you want the benefits, you’re willing to put in some work and time and effort. You’re willing to be the face for a little while, but yeah, your real goal is to have this move forward and hopefully get kind of a downhill slope. You’ll start moving in and it’ll take on a life of its own.
Amitai: I hope so, and I hope that it is not about me, and so far I’ve been very lucky in that regard. So one of the first things that I did as I was thinking about how … not only who do I invite, but how do I invite them? Because I have a lot of power in that moment to do something more persuasive or less, or give them better information to make their decision or worse. So I thought what I should do is write down a list of questions that our users would probably have if we’re ever so lucky as to have users and that this developer that I’m asking to join me will probably have as they consider whether to switch from solo individual developer mode to something collaborative.
Amitai: What is this collaboration going to look like? What are the goals that I have in mind for this project? What are things that are explicit non-goals that I’d like to avoid? What are some features I imagine adding? What are implementations of those features that I wouldn’t want to merge? So we can get a picture of what we would say no to. What is a good pull request look like? What are the team norms for? What makes a pull request good? What are we looking for in a review and after how many reviews do we merge it? Then something like a roadmap in particular, because we’re in a forest of, if you can imagine 20 year old idiosyncratic C code, there are 8,000 things that can’t stay the way they are.
Amitai: But there are a million paths out of this forest that are suboptimal. So for example, let’s say one really obvious thing, if you look at the C code and you’ve ever looked at any other C code, you’ll notice that this uses a lot of externs instead of using includes of the system libraries, and that’s because in 1998 the system libraries might’ve been lying to you. They were notoriously buggy on various systems that qmail could run on.
Amitai: And so one thing that looks like low hanging fruit here with my forest metaphor is let’s change those externs to includes. That might be a really easy change to make and it might make us feel a little better, but it would put a pretty significant portion of our users at some risk that we don’t want to put them at. Because everybody who runs qmail has to manage their own patch farm and if we do something invasive that might make their patches not apply before we’ve implemented the features that make their patches not necessary anymore, then they’re going to be mad at us and are going to fall off the train and we’re not going to have them anymore.So as much as we’d like to change those externs to includes, that’s for much later, for example.
Marcus: This might be the only episode that we use the word patch farm on, but I hope not. I want to go back to something you said a minute ago, the idea of the power you have in how you invite people, because that just kind of struck me that I feel like oftentimes as managers and leaders, we don’t do as much inviting as we should, or we do the phrase I heard the other day, someone, we volunteer people, well you’ve been voluntold and that is obviously we’re applying a lot of pressure to do something. Even though we’re couching it like a question, would you please volunteer to take out the trash right this minute? That may not really leave you with many choices.
Marcus: So when you see teams working together and teams doing it well, why does an invitation look like to join a project? Let’s say we move this context within a company, within the employment context, do you see that managers and leaders are inviting people to different levels of participation or different kinds of participation? And what does that kind of invitation look like? That is really a generous, true invitation, not just volunteering someone.
Amitai: Yeah, I’ve certainly been voluntold more than once. I definitely have an advantage with this project in that I hold no positional power. In fact my expectation was that nobody’s going to be interested in this. So, if I have a chance it is going to be about persuasion, and I did individualize my invitations to each of the people that I invited based on what little I knew about them, about their work or about their style or what have you. In a business context, there’s always a backdrop of what if I say no.
Marcus: Right.
Amitai: Does my money stop coming to me, did my benefits cease?
Marcus: Can I make my mortgage?
Amitai: Yeah, things like that. The best intention of managers and tech leads may have none of that in mind and it may not even be in their power to execute, but the person hearing it may hear it anyway and so really I had it easier. There was nothing like that riding on it for the people that I was asking, and for me it’s easier to do the asking when I know that nobody’s going to be able to hear me that way. It’s a lot harder in a business context.
Marcus: Yeah. My guess is there aren’t many MBAs that talk about the disadvantage of power but I think you basically said that, you said I had the advantage of not being in authority and so therefore, my request was truly for one person to another, not as someone who could threaten or coerce or anything else. So I guess the thing that is sparking for me right now is that as managers are thinking about how can they get more participation from their team, how can they get more motivation and buy in? These are all words that everybody seems to want more of.
Marcus: How often are we thinking about persuasion and about invitation and about leaving your titles at the door? I recognize that some of you might say, well, that can never be done. Maybe that’s true, but my guess is that if you create a place, a virtual place, a physical place where you say, we’re going to walk in and everybody’s going to be the same peer for the next hour and these are the rules I’d like us to try, they may not trust you right up front, but maybe in time if you practice having a peer dialogue, maybe those things are possible.
Amitai: This is right on point for what I do professionally. I’m an independent software development coach, a technical agile coach. I discovered in a training course with Alistair Cockburn, a few years ago that one of my go-to techniques is that when a situation is a little confused, one of my defaults is that I will seek to lower my perceived social status. This only gets more and more important as I get more and more invited as an outsider with a name and some status because I cannot do the influencing that I want to do if I’m a big shot that people are supposed to do what I say. This is a double edged sword, like my whole career, until I became a consultant, I wished I would get listened to more.
Amitai: Now it happens and I have to be careful because things that I might say like off-handedly half thought through ideas, I didn’t mean go enact my half thought through idea immediately, but sometimes that happens and that was what I wished for. But I have to be more careful about what I say. Another times it doesn’t matter what I say, people see me as an outsider who is the 17th to visit them and they will outlive me too. Let’s just grin and get past this guy until we can go back to what we know. So for me, a go-to technique is … I genuinely don’t know more about what’s going on there than the people that I’m sitting with. I think I make it clear to them, one of my favorite techniques is to … we’re looking for our first idea and the room is nervous because I’m there.
Amitai: I’ll go first, I’ll give an obviously not good idea. Now we’re starting, now again, it’s safer for me to do that than it is for somebody who works there to give an obviously not good idea. They might think somebody is taking them seriously and their ideas aren’t good and they should stop having ideas, but I can safely do it and that starts things rolling and it gives them an idea about who they’re dealing with when they’re dealing with me, that allows us to relate to each other a little better.
Marcus: Amitai, where can people find out more about this project?
Amitai: This project can be read more about at notqmail.org.
Marcus: Where can people find out more about you, the marvelous services you have, maybe how to hire you and fix their problems.
Amitai: Sure. I have a three minute podcast, it’s called Agile in 3 Minutes. I have a personal website with lots of tech blog kind of posts and occasionally coaching posts, that’s at schmonz.com, S-C-H-M-O-N-Z and my consulting can be read about at latentagility.com.
Marcus: Amitai, thank you so much for being on the show today. I’d love to do this again when you’re a little farther down the road and hear how things are working with the notqmail team.
Amitai: This was a pleasure and that will be too.
Announcer: Thank you for listening to Programming Leadership. You can keep up with the latest on the podcast www.programmingleadership.com and on iTunes, Spotify, Google Play or wherever fine podcasts are distributed. Thank you for listening and we’ll see you next time.
This has been a HumblePod production, stay humble.
You may also like...
