Don’t Be That Guy

Nobody likes the “brilliant asshole”.  Don’t be either of those things.

Doesn’t matter how much good work you do if you’re an insufferable jackass.  You might be right or have good ideas but nobody’s going to want to work with you.

Doesn’t matter what you have to say or what you have to show for your efforts if people hate your guts.  You might do something fantastic; doesn’t mean a damn thing if no one gets behind it.

Don’t be that guy.

Common Sins of Systems Administration

I feel as though I’ve spent enough time doing Systems Administration that I figure it’s time to vent a bit and point out some common fallacies and reasons why these are anti-patterns for both successful SysAdmins as well as overall health of tech organizations.

“Just restart the service/host.”

If companies wanted this kind of response to a problem, they would just invoke an event-driven service to restart services or hosts when a failure is detected.  This just masks the problem.

Instead:

  • Ask “Why did this fail?”
    • If you’re not asking why, you’re behaving much like the oft-maligned “Windows Admin” or “Bastard Operator From Hell (BOFH)”–restarting services or servers and not digging in deeper.
  • Look at logs.
    • If you’re not looking at logs for a service or a host, you’re missing a huge part of the work that makes Systems Administrators one of the most important roles in a technology organization.
  • Look at service and/or host statistics.
    • Back up your assertions with data, otherwise you’re just conjecturing.

“It’s not important/it’s intermittent.”

Bullshit.  If it’s important enough to page someone, it’s important enough to debug and find a way to keep it from happening again.  If it’s intermittent, then there’s a cause.  Ignore the symptoms, find the source.

“I’m too busy with other stuff.”

Again, bullshit.  If you’re too busy, say so.  It’s your job to tell someone (your manager, your peers, your employees, whatever) that you’re over-subscribed.  Anything worth doing is worth doing right and devoting as close to your full attention to as possible, so speak up.

“I don’t know how to do this.”

Then you’ve just been gifted with a learning opportunity.  The best SysAdmins use learning opportunities rather than avoid them.

  • Read documentation.
  • Look at logs.
  • Debug the problem.  Try to duplicate it.  See how it differs given different conditions.
  • Understand the services and hosts that you’re responsible for.
  • Be thorough and verbose in your investigation.

“This is never going to be fixed, so why bother?”

Your entire job is predicated on the notion that you’re supposed to help provide context to decision-makers around impact of bugs, flaws, and failures.  If you’re not doing it or aren’t interested in doing it, then why are you even working?

  • File bugs/stories/incidents.
  • Provide data to back up your assertions/findings.
  • Dig in.  Look at logs and statistics for commonalities and outliers.
  • ADVOCATE.
    • If people aren’t listening, take it up to the next level.  Find the Product Owners or Product Managers.
    • Explain why it’s a problem and what they could do to resolve it.
    • Talk about the cost of the problem in terms of payroll hours wasted (this generally gets people’s attention).

These are definitely common sins, but this is by no means a complete list.  What do you think?  Have any you’d like to share?

Mental Health & The “Technology Industry”

My relationship with the “technology industry” has been contentious in many ways.  I’ve been diagnosed with and am currently undergoing treatment for PTSD, struggled with depression, and have intermittently contemplated suicide.  Constantly fighting with myself, battling Impostor Syndrome, and pushing back against my own unrealistic expectations of things is as exhausting as you might imagine.

“The industry” has talked at-length in recent years about figuring out ways to reduce or eliminate burnout, and it’s been helpful as far as exposure is concerned.  What hasn’t improved is the actual effort around improving things.  Organizations can talk all day about giving people more breaks, encouraging contributors to take more vacation time, refer them to specialists, and add more “perks” like nap rooms, alcohol, spaces and activities to socialize, but addressing the root of the problem requires getting into the sharp, thorny thickets of human emotion and understanding the human costs of doing business–a subject that few organizations seem interested or able to engage in.

I had a recent conversation with someone that travels in the same circles that I do in the New England technology industry after they posted an article about an Uber engineer that recently committed suicide due to work-related stress.  I took great pains to point out to them that the tech industry does a great job of increasing visibility–but that’s generally where the effort stops.

Let me be clear: visibility does not solve problems.  People solve problems.  Engineering problems, people problems, organizational problems, global problems.

Just because you know someone is close to burnout doesn’t mean that their situation is magically going to improve.  It requires communication.  It requires an immense amount of trust.  Effective communication and trust require vulnerability.  Vulnerability veers into territory that most organizations, whether explicitly or implicitly, generally steer individuals away from.  While organizations might make “diversity” and “inclusivity” core pieces of their mission statements, the actual work that ends-up going M.I.A. is the actual skills around having difficult conversations and how to provide support for each other.

In my own experience, the organizations I’ve worked for have actively avoided all of these conversations and haven’t built a reasonable framework for disclosure or discussion.  A significant number of the experiences I’ve had so far basically boil-down to some variation of the phrase “don’t let it become a problem”, implying that not only do the organizations not care, but that there are also potential consequences if the individual “can’t keep a lid on it”.

Organizations make it easy and expedient to avoid communicating with each other about the things that make work-life balance possible.  On top of that, there’s always the implication in larger or cut-throat VC-funded organizatons that the point of the business existing in the first place is (ostensibly) to make money.  “Human costs” of doing business are so poorly defined and nebulous that making them part of the overall calculus often results in nothing changing.  Unless an enterprising soul is able to tie the costs of “toil” (in the parlance of Toyota) or “unplanned work” directly to a revenue stream or cost-center, the business, to be utterly blunt, just doesn’t give a damn.

“That’s what we hire you for!  You do work, we pay you.  End of story.”

All things being equal, it’s been implicit rather than explicit that there’s a core belief in every tech organization: “if you can’t hack it, you have no place here”.  Which further implies that there isn’t support availble internally for problem solving, training, or even simply to “check something out with someone”.  I’ve completely lost count of how many times I’ve wanted to walk over to a Product Manager or an Executive and ask them why de-prioritizing a change to reduce toil was a good idea.  Or how the calculus of employee churn compares against expected costs in a given group or department (hint: it doesn’t).

How do you re-engage an employee who’s lost in their own head?  How do organizations make it okay for someone to have feelings and work on their problems while still providing value as a contributor?  How do we provide real culture in organizations rather than cheerleading sessions and frivolous, alcohol-laden socialization?

We need to create connection.  We do that by helping everyone to understand their inherent value and to really appreciate their contributions, no matter the size.  When we are able to help connect people together and help them realize that the code or the product is just a MacGuffin for connecting people together in pursuit of something greater.  Whether that’s self improvement or improvement of a product or the world writ-large, that’s what we really need.

What are the costs of an employee burning out and leaving?  What is the impact of constant, interrupt-driven events on engineering teams?  What, if anything, can help drive more meaningful interactions between individuals and teams to help take emotional risk from lethal to livable?  How can we take suffering, whether silent or not, out of the picture and instead figure out how to uncover an individual’s “joie de vivre”?

We could start by inculcating a culture of “acceptance”.  Rather than telling or implying to someone “shape-up or ship-out”, we can make room for them.  Ask them what’s going on.  Talking about our personal lives at work is only problematic when the organization makes it a problem.  When fear overrides our common and innate ability to empathize.

To reiterate: businesses don’t like risk or liability.  Personal lives are rife with situations that organizations, whether out of risk-avoidance or cowardice, tend to want to avoid any discussion or mention of.  Human life is exceptionally complicated, with some being more complicated than others.  If we’re not able to talk about what is creating disharmony in our lives, whether it’s personal or professional, like actual adults, then we’re missing the point of working together in the first place.

We could continue progressing through improving communication.  Telling someone that their decisions are creating an unfair burden or that they felt as though they couldn’t speak-up is key to improving the working life of anyone in technology.  Being able to talk openly and honestly about the causes of their stress, whether it’s too much work, feelings of powerlessness or uselessness, or a lack of improvement, being able to respond to and mean what you say goes a long way.  Being able to say “I can see where you’re coming from”, “I didn’t know this was a problem”, “I’m sorry things aren’t going well”, or “I want to help you, tell me how I can help” can help many people in very meaningful ways.

Lastly, we need to start building a culture of “work to live” versus “live to work”.  The technology sector lionizes the hard-drinking, quick-developing, move-fast-and-break-stuff, up-til-3am-saving-this-client’s-butt type of employee, but does little to really incentivize or provide meaningful feedback for people who contribute in a more healthy way.  If the technology industry were serious about work-life balance, there’d be less emphasis on balance-sheets, PNLs, and work-hard-play-hard engineering culture, and instead would make work-life balance their core, internal mission.  Unofficially, where I work currently likes to play with the phrase “No Capes”, but we’re still a long way off from making it a reality.

The technology industry needs to more clearly understand that there is more to be done.  It’s not enough to simply acknowledge it and move on.  It’s not enough.  Organizations need to implicitly, and intimately, understand the impact of work-related stress as it relates to mental health and the health of the broader organization.  Where organizations seem to fail most is making it safe to even start the process–so I encourage you to bypass the organization entirely.

Start with your peers.  You know at least one who probably has a lower threshold for frustration or someone who’s probably having a tough time with things.  If you really care about them, ask them how they’re doing.  They just might surprise you and tell you that they need help with someone that you’re good at or have experience with.

You probably then want to start thinking about your group or your organization.  How can you do better to support people that might be suffering in silence?  How can the group or the organization make it easier for people to get help and to sound the alarm when they’re overwhelmed?

Finally, what can the organization or the business do to make their operations more sustainable?  Are the human costs of technical debt or procedural gaps justified when the quality of life for employees is compromised?  I’d argue that no cost is too high in maintaining a high quality of life for any employee, regardless of position or function.  But the important part is starting the discussion.

We can all do more to help improve the lives of our peers, our friends, our coworkers, our communities, and our loved ones.  If we can at least start the conversation, then we might just be able to improve the lives of the ones who need help the most, but might not have the voice or the words to ask for it themselves.

Mental Health in *Ops

What are the first things that come to mind when you think of the words “mental health”? Straightjackets? Pills? Psychiatrists and leather couches? If they are, you might not need to place all the blame on yourself. Western culture, to this day, still manages to maintain a deeply-entrenched stigma around the notion of mental health or disorder; the very words appended together evoking all manner of distinctly negative connotations. We’ve tried as a culture, in many different ways, to describe it: disorder, illness, disease—antonyms of what would generally be considered “normal”. As a direct result, individuals in Western society are now working to find ways to engage in the Herculean task of ending the stigma of mental health and understand its impact on individuals, groups, and society more broadly.

What do I mean by “*Ops” and what does mental health have to do with it? More than you might think. I use the phrase *Ops to describe all Operations groups and individuals rather than just a single subset of them: NetOps is not DevOps, DevOps is not SysOps, SysOps is not SysEng, SysEng is not NOC, etc. However, all of these title are more generally discussing Ops as a whole, thus the idea of *Ops. It’s inclusive and acknowledges the different specialities and skill sets that many *Ops people are familiar with but might not have a focus or interest in, and I think it more accurately represents the diversity of roles within the Operations discipline.

More importantly, what does #HugOps mean? More than a Twitter hashtag, it’s a set of core beliefs and ideas whose time has come. The wonderful people over at PagerDuty wrote a short-ish blog post about #HugOps, empathy skills for DevOps, and its impact on team members as well as the broader organization.  It means embracing the whole of Operations (even the “Type-2 Fun” parts of it), engaging in and improving sustainable Operations practices, and just generally being an empathetic human being.  Sustainable Ops practices aren’t just focused on business continuity and velocity, they’re also concerned with things like quality of life, encouraging personal growth, and skill development.

To the point then: why talk about mental health and Ops in the same breath?  Because they’re inextricably linked, bound at the atomic level by the sheer fact that we are humans operating complex systems.  By inference, complex systems are prone to failure, and as a culture we do a terrible job of decomposing problems into their atomic parts without laying blame on someone or something, whether conscious or not.  When a team engaging in DevOps practices produces a service on which your system or service lives on encounters a failure of some type, of course you’re going to have a reaction.  What’s important is understanding the reasoning behind it and decomposing it so it becomes less of a people issue and remains simply an issue.

Why is it hard to talk about mental health in a work setting?  When we view the question through the lens of traditional (borderline “legacy”) organizations and structures, we often come to the same conclusions: that these issues are “thorny” and are better left to silence.  What we are learning instead is that when we leave these topics in the shadows, they fester and grow into pernicious and poisonous things.  In many cases, HR guidelines in a lot of legacy orgs reflect this viewpoint: feelings are thorny and difficult, the only thing that matters of productivity.  From that same perspective, the notion remains: “So long as your work is getting done and we are paying you a fair wage that we are legally bound to pay you, we don’t care.  Just don’t rock the boat.”

I once had a friend tell me, “HR isn’t there to help you or protect you.  They are there only to protect the business.”  I think what that person was trying to convey was that HR in legacy orgs isn’t (generally) there to be your friend or to help you.  It’s there to ensure neither the business nor you are subject to a lawsuit.  It’s been my experience that those self-same orgs are among the worst jobs I’ve ever worked at.  I worked at a Fortune 50 company many years ago where employees and contractors alike were afraid to talk about personal matters anywhere in proximity to the building for fear of being put on a short list for layoffs once the cycle came back around.

If your organization sounds even remotely like this, I advise you to run.

The notion that these types of guidelines, rules, and groups exist with the idea ingrained that employees are incapable of being functional, rational, empathetic adults at their place of work not only speaks to a level of organizational naivety, it is plainly insulting to anyone who has even a modicum of self-respect and self-awareness.  To that, I call bullshit.

To the above point: what kind of impact does that have on an individual?  Maybe someone who is in struggle at that very moment, maybe someone adjacent to another who is going through a difficult time in their life, has a chemical imbalance, or is a survivor?  It perpetuates the stigma of mental illness and further isolates individuals who might otherwise be strong contributors to both productivity and growth within a team or an organization.  When silence is the only option, that person’s life suffers in a number of ways:

  • Decreased motivation
  • Decreased cognitive function
  • Loss of appetite/sleep
  • Mood variability (depression, mania, bipolar-like symptoms, trauma, etc.)
  • Substance abuse

The list goes on.  To bait the hook even further: why, in two-thousand-and-fucking-sixteen are we still fighting to have this conversation?  I’m not sure I’m able to answer that question without talking more about my own experiences.

I started at my current job a little over two-and-a-half years ago.  In that time, the customer base and the fleet size that we operate has increased by an order of magnitude, and so had our problems.  When I started there, I had just left a job where the manager I was working under had coaxed three distinct panic attacks out of me in a single day.  I turned in my two week notice that very day and nearly refused to do the exit interview because I had nothing positive to say about the company or the person I had worked under.  When I went to the new gig, things were mostly okay.  I contributed to documentation, improved the JIRA dashboards that we used, started putting together a unified shell command structure that would eventually be used by the entirety of the rest of the Ops team… things were going well.

What I didn’t know was that in my personal life, I had failed to exercise the primary rule of Ops: Fix the problem, not the symptom.  I had left one job for another, but in the end I traded one set of stressors and symptoms for a completely different and drastically more complicated set.  I was already having a bad time of things due to depression, but laying impostor syndrome and a large amount of interrupt-driven work on top of the personal issues I was already experiencing was the straw that broke the camel’s back.

I began to see a therapist.  I was already familiar with a lot of self-help and motivational methods and books, but what that person helped me to see was too large for me to ignore: I suffered from non-combat-related Post Traumatic Stress Disorder, and still suffer from it to this day.  (Side-note: the fact that I have to specify non-combat-related screams loudly that we have a long way to go as a culture when discussing mental health.)

What would take me many months and a complete disintegration of a romantic relationship was seeing how the symptoms pervaded every part of my life, from intimate relationships all the way out to professional relationships.  My work suffered.  I was on-call often and being paged in the middle of the night only contributed to an inability to cope with “minor” stress.  Sometimes I’d be paged over a hundred times in a single shift.  I started to lose motivation, not just at work but in my personal life as well.  I stopped going to the gym, I stopped caring about personal relationships, and I began a general spiral.

That’s when I decided enough was enough.  I took a month-long road-trip and got away from the stress so I could try to think critically.  I found that I could sleep at night again.  I started to remember people’s names and faces more easily.  I drank a lot less.  I also took probably one of the most important steps out of everything I’ve done for myself in the last few years:

I explained to my management team that I was undergoing therapy and explaining everything that that entailed.  It wasn’t the most comfortable conversation, but conversations that make room for growth and reflection seldom are.

What has followed since that conversation has been nothing short of eye-opening for me.  I’ve found that there’s a renewed sense of empathy and camaraderie among the Ops people I work with currently.  Rather than trampling each other in stand-up conversations, we are more apt to lay out our thoughts in emails and chat and regroup after emotions have dissipated.  I notice more people are asking how I am doing and if there’s anything that they can do to help–and vice-versa.  I now find myself asking coworkers that are visibly stressed if they need me to take something off their plate for them or talk with them about it.

I’ve learned over the course of this entire experience that there are non-negotiable things that every person has to have available to them:

  • An explicit option to be real with their management and their peers
  • The specific ability and option to set boundaries and expectations with everyone they work with
  • The ability to push the Nope Button/pull the rip-cord (and have that be absolutely okay)

That being said, there’s a lot of room for improvement in the *Ops community from the individual level all the way to the organization level.  “Making room” for people to be able to talk about the difficulties they’re encountering in their lives and how it impacts their work is just one small part of it.

Be empathetic.

  • “Hey, I’ve noticed you’ve been a bit out-of-sorts lately.  Want to talk about it?”
  • “I want to help you succeed.  If that means adjusting your responsibilities here at work so you can be less stressed and healthier, let’s talk about it.”
  • “I understand you’re stressed out.  I feel the same way sometimes.  Would talking about some of it help?”

If HR is an impediment, get them out of the way.  If HR policies or the team is hindering your teams’ ability to be real with each other and their management, it might be a good time to address that.  If people are afraid to be empathetic, rational, thinking adults, then I would think that that would carry over to their everyday interactions with other people.  Culture is both organic (bottom-up) and codified (top-down), not just one or the other.  If what you’ve codified doesn’t match-up to what has been cultivated organically between people, then maybe it’s time to ditch the handbook and instead opt for real interaction.

If you’re a manager and you’re thinking “my team doesn’t have any of these problems” or “this doesn’t apply to me”, then it most certainly applies to you.  Symptoms of mental health might not be visible, but you might recognize them as your “intermittent” or “low” performers (which, by the way, is a terrible thing to categorize someone with).  Personal life circumstances might make the work-life more difficult for some people, and can be exacerbated quite easily.  Losing team members due to stress, burnout, or mental health issues can have a ripple-effect throughout the rest of the team, maybe inducing others to leave and seek better (or less stressful) opportunities.

Educate, educate, educate.  There is literally no excuse in this age of information to not know the signs of someone in struggle with mental health issues.  The resources available are things that we use every day of the week: Google, Wikipedia, HealthMD–the list goes on.  If in doubt, just fucking ask.  Genuinely give a damn about your team’s happiness.  If you don’t have something you can do immediately to make the situation better or easier to handle, then at the very least empathize.

  • “I hear you.  I’m sorry you’re struggling right now.  I don’t have any good answers or anything I can do right this second, but I want to make things better.”
  • “Thanks a lot for telling me.  I appreciate your honesty and vulnerability, and I’ll do what I can to make things right.”

Finally, what can organizations as a whole do to help make things better?  It’s actually a lot easier than you’d think.  “Good culture” is more than free beer, tabletop games, catered lunches, and open collaboration spaces.  If these are your main efforts or focus on cultivating great company culture, then you’re doing it wrong.  If you want to attract “10X Engineers” or “Rockstars” who will leave in 6 months because they got bored, then sure–do that.  If you want to attract people who will stay, who believe in “the mission”, and want to provide the best customer-facing and work experience possible then do the work.  Learn about your employees, figure out their wants and needs, and do what’s in your power to make them a reality.

Feedback loops seem to be overlooked in some places and over-utilized in others.  Here’s a simple way to do it if you can’t change HR policy or company culture in the short-term: anonymous survey forms.  By eliminating the stigma of speaking up about frustrations and letting people contribute to the feedback loop with uncensored input, it can give leadership great insight into where gaps exist and how best to resolve them.  Tossing proverbial “grenades” into conversations and stepping back to moderate the discussion can also be a great way to give team members a sense of empowerment and ownership over issues they feel strongly about.  I’ve found personally that when someone feels strongly about a specific issue, they’re going to take it and run with it until they’ve reached some kind of satisfactory response or at the very least an understanding of the geometry of the problem.

Lastly, this should go without saying: be a resource.  Even if you’re not in management, you can be a vocal advocate for someone in struggle.  Cheerlead for them when they do well, counsel them and step into the thick of it when something goes wrong.  Ask the Five Whys, be provocative in your questioning, make recommendations (not mandates).  Give them a sense of empowerment and ownership.

In short: be the kind of person that you’d want others to be for you.  Lead by example.  Short of that, find somewhere that you can be honest and be yourself without apologizing for being human.

Exploring Ansible

At the behest of a friend in the DevOps community in Boston, I’ve been trying to give Ansible a go.  In thinking about this, I’ve also realized that I’ve stumbled across a few problematic areas in getting a basic Ansible setup that I hope to help others avoid by documenting them here.

First and foremost: I’m not a developer by any means.  I’m a glorified Systems Administrator by-day and a first-responder to outages generated from the aforementioned systems during the weekends.  What I’ve come to realize over time is that in order for me to progress at all, I need a better understanding of how more modern, dynamic services, APIs, and the assorted building blocks of the modern service-oriented web work.

The way I chose to go about this initial bit of discovery is pretty straightforward: I’m using a Vagrantfile to talk to DigitalOcean and kick off a droplet, and then I am using that droplet as a target for some test Ansible commands.  There are probably much easier (and frankly, more interesting) ways to do this, but this is what I know so I’m starting here.

First thing I did was get a directory set-up for this little test:

mkdir -p ~/ansible
cd !$

Vagrantfiles are pretty easy to grok and maintain.  If you can read and follow along with basic Ruby syntax, you’ll be able to modify what you need.  I started by initializing the Vagrantfile:

vagrant init

Once that completed, I had a directory structure that looks similar to this:

14:15:15-thomas-jamesgoin~/ansible$ ls -lah
total 32
drwxr-xr-x 6 thomas-jamesgoin staff 204B Apr 26 14:09 .
drwxr-xr-x@ 76 thomas-jamesgoin staff 2.5K Apr 26 14:09 ..
drwxr-xr-x 3 thomas-jamesgoin staff 102B Feb 14 22:12 .vagrant
-rw-r--r-- 1 thomas-jamesgoin staff 4.1K Mar 13 11:40 Vagrantfile

And then came the fun part: figuring out how to interact with DigitalOcean’s API.  From my current job, I have a pretty basic understanding of how tokens, APIs, and API keys worked, so I started looking around for documentation on how I could talk to DO without doing the GUI thing.  I stumbled upon the Vagrant plugin vagrant-digitalocean, which thankfully only requires a quick Vagrant CLI command to install:

vagrant plugin install vagrant-digitalocean

After that, I located docs on How To Use the DigitalOcean API v2, which made it easy for me to get a token and talk to the API.

Point of note: the way that I’m doing this operates on the assumption that I have a separate SSH key pair for use with DigitalOcean. This is not only good security practice, but also a great way to protect your primary SSH key pair. There’s plenty of documentation on how to generate an SSH key pair on the internet, so I’m not going to cover it here.

Using the docs on the GitHub page for the vagrant-digitalocean plugin under Configuration, my Vagrantfile looked something like this (I had to prune commented lines out of the below example due to character limits in blocks with hosted WordPress):

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  config.vm.box = "digital_ocean"
  config.vm.network "public_network"
  config.vm.hostname = 'dropletname.example.com'
  config.vm.provider :digital_ocean do |provider, override|
    override.ssh.private_key_path = '~/.ssh/digitalocean_vagrant'
    override.vm.box = 'digital_ocean'
    override.vm.box_url = "https://github.com/smdahlen/vagrant-digitalocean/raw/master/box/digital_ocean.box"

    provider.token = '(TOKEN)'
    provider.image = 'ubuntu-14-04-x64'
    provider.region = 'nyc2'
    provider.size = '512mb'
  end
end

Testing this was simple:

$ vagrant up
Bringing machine 'default' up with 'digital_ocean' provider...
==> default: Using existing SSH key: Vagrant
==> default: Creating a new droplet...
==> default: Assigned IP address: (REDACTED)
==> default: Rsyncing folder: /Users/thomas-jamesgoin/ansible/ => /vagrant...

And voila! A working DigitalOcean droplet instantiated from Vagrant!

So that was the more interesting part. Ansible installation and execution itself was the far more pedestrian effort (not sure why it asks for
sudo, but whatever).

$ sudo pip install ansible

Once that was completed, I took the IP address in the output of the vagrant up command and added it to a hosts file in my project directory. My directory structure now looked like this:

14:30:48-thomas-jamesgoin~/ansible$ ls -lah
total 24
drwxr-xr-x   6 thomas-jamesgoin  staff   204B Apr 26 14:24 .
drwxr-xr-x@ 76 thomas-jamesgoin  staff   2.5K Apr 26 14:24 ..
drwxr-xr-x   3 thomas-jamesgoin  staff   102B Feb 14 22:12 .vagrant
-rw-r--r--   1 thomas-jamesgoin  staff   3.8K Apr 26 14:24 Vagrantfile
-rw-r--r--   1 thomas-jamesgoin  staff    15B Apr 26 14:02 hosts

However, I had another problem: Ansible defaults to reading /etc/ansible/hosts, which isn’t really convenient for folks on OSX, so I added this to my ~/.bash_profile file:

###
# Ansible stuff
###
export ANSIBLE_HOST_KEY_CHECKING=False
export ANSIBLE_HOSTS=/Users/thomas-jamesgoin/ansible/hosts

Now I was ready to test an ad-hoc ansible command:

14:40:35-thomas-jamesgoin~/ansible$ ansible all -m ping -u root --private-key=~/.ssh/digitalocean_vagrant
(REDACTED) | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

I’d discovered that with the way DigitalOcean currently provisions droplets, you unfortunately use -u root to run anything as the droplet has no user-accounts installed by default. There’s likely ways to ensure this is the case via Vagrant, which I’ll probably end-up investigating later. Similarly, --private-key= was required here because I haven’t quite figured out how ssh agent forwarding works in this whole setup… but at least I’m a few steps ahead of where I was when I started.

The next few steps I’ll be working on is provisioning a non-root user as part of the Vagrant bootstrap process, figuring out how to boot multiple droplets from the Vagrantfile, and how to do a simple app deployment. More to follow!

Dismissing Evidence

It might seem easy to dismiss problems in any group as rabble-rousing or hyperbole.  What’s more telling is the dismissal or lack of focus on the problems rather than the acknowledgement.

On the other hand, acknowledging problems and pressing-on anyway might be the best way to show others in the group that the focus might be on problems they can’t see… that really might not be problems at all.

A person goes to the doctor with chest pain and a general sense of malaise.  The doctor tells them that they have lung cancer and maybe a few years to live with a small chance at another decade or two if they change their habits and address the fact that they have a condition.  What do they do?

Someone who sees (and trusts) the evidence, the statistics, and the interests of the people they’re asking will likely stop smoking, start taking their health seriously, and start advocating against the very things that hurt them in the first place.

Someone who inherently distrusts everything and is unable to see the truth probably won’t care.  Chain-smoking, binge-drinking, obscene fast-food intake; speeding it up is the only thing that makes sense when outside evidence or results directly contravene the focus.

The same type of scenario plays itself out every day in organizations where legitimate information, discussion, debate, and dissent are either muzzled or overridden by profit or prestige (I would argue that it’s actually the sin of pride, but that’s a bit too much religion at work).

Technology has already had its soul rent asunder twice before; if organizations and leaders within those organizations fail to understand and internalize the lessons of the previous boom-bust cycles of the DotBomb era, I would think that in this age of on-demand contracting and instant entrepreneurship would give them pause.

What if I decide to take in all available evidence, have a transparent discussion, and get a game plan together, and actually become accountable for the decisions being made?”

What if I allow myself to invest my trust and faith in the people that I’ve chosen to hire and work with and give them the necessary space to bring their creative energy to-bear on the issues at hand?”

What if I choose to be a little more vulnerable with short-term investment of time or capital in something that might not work, but might make all of our lives better in the long-run?”

“What if”, indeed.  This is where I personally see a lot of organizations fall apart.  Too many what-ifs, too much uncertainty.  The need to profit from every keystroke; every moment spent reading text, every bit, byte, bloop, bleep, and click.  When that need overrides the desire of why the business became a business in the first place, that’s when the magic fades.  That’s when decision-making turns into decisions about margins and when human investment (emotional, physical, and spiritual capital) becomes tertiary.

That’s when people make choices on their own and sometimes even choose to leave.

Rage-Driven Ops (or “Don’t drink-and-type”)

People can be pretty bad at communicating.  We are often so buried under various types of work that it becomes a struggle to “keep your head above water”, and often treading that water tends to promote anger, resentment, and frustration.  I’ve noticed a pretty common theme among most places I’ve ever worked:

Every place I’ve ever worked has failed to allow or create space for people to be able to vent their frustrations and for people to ask constructive questions.

For example, when we talk about clustering or provisioning where I work now, those conversations tend to be accompanied by wild gesticulation, increases in vocal volume and intensity, and indirect finger-pointing.  What tends not to happen during these arguments is communication.

Asking the Five Whys when you’re in the thick of things can help you get a clear perspective on things:

  • “Team X is a bunch of retards!  They don’t know how to do anything right!” “Why?”
  • “Their tickets come in with missing or bad information!” “Why?”
  • “They don’t know what to fill in or where to get that information!” “Why?”
  • “They don’t know how to use their tools or to when to ask someone on their team for help!” “Why?”
  • “Maybe it’s a documentation or training problem, I don’t know.” “Maybe you should talk with someone from Team X and ask them to help make this better?”

This is a rather oblique example, but seeing the process in action and being able to reach beyond yourself and beyond the limits of your station to improve things is probably among the most valuable things you can do in an organization.  Some things that technical people may not keep in mind when they interact with other teams:

  • Your experience and knowledge != their experience and knowledge.
  • They are not stupid (if they were, they likely wouldn’t be where they are).
  • They are probably experts in their chosen silo, just as you likely are in yours.

Clearly define your knowledge gaps, address the process failures, and probably the most important point of all: have empathy.  Otherwise, why are you doing the work that you do in the first place?