Quality Sense Podcast: Erika Chestnut on Calendly's Test Automation Strategy

In this episode of Quality Sense, Federico has a conversation with Erika Chestnut, Head of QA at Calendly. She started her career as a developer, but after complaining about quality a few times too many, she found herself learning more about software testing. Today she is a Quality Coach, mentor, consultant, and keynote speaker. Listen to the episode to learn about how test automation plays a role in Calendly’s test strategy, and, as with every guest, her advice for testers on good habits, recommended reads, and more.

Episode Highlights:

• How Erika first learned about testing and why she is passionate about it
• Building a solid foundation for automation
• Pros and cons of having the developers at Calendly drive test automation
• How they align the developer’s automation effort with non-automated testing driven by the testers in order to cover what’s important at the different layers and to avoid duplicate tests
• Involving DevOps and the customer experience area to perform stakeholder testing

Relevant Links:

• Connect with Erika on Twitter: 
• Visit 
• Make your scheduling life easier with 

Listen Here:

• Listen on 
• Listen on 
• Listen on 

Episode Transcript:

Federico:

Hello, Erika, how are you doing today? I’m so happy to have you here on the show.

Erika:

Thank you. I’m so happy to be here. I’m excited to chat.

Federico:

Yeah. So my first question for you; being the Head of Quality Assurance in Calendly, how does it feel to be working on a product that probably your friends and colleagues are using every day?

Erika:

That’s probably the coolest part is it’s not only a product that a lot of people I know encounter and more and more are encountering, but it’s a product that I get to use that I personally use. And it’s kind of a conversation starter, even when I’m just talking to people in my own personal world or business and stuff like that and they send me a Calendly link or if I send them a Calendly link, but especially when they send it to me, I’m like, “You got brownie points.” There’s a lot of pride in that. And also hearing their feedback about how it’s going well and why it’s helping them. Or if they’re like, “Hey, can we get this added in?” Or, “I’d love to see this area improve.” That’s always great to hear because I know people are interested in joining.

Federico:

I can imagine that. It’s also this idea of eating your own dog food or something like this, right?

Erika:

Yeah.

Federico:

So you have another perspective with the product that you are building. That’s cool.

Erika:

Exactly. Exactly.

Federico:

So tell me how you started working in software testing because I checked your LinkedIn profile, it’s pretty impressive, your career. So I want to know how you ended up working in testing.

Erika:

I complained about the quality. I started out as a developer and I had worked at a few places. I was at a marketing agency, an advertising agency, and I’d been there for several years as a developer. And I moved up into leadership and we were having quality problems. We didn’t have a quality department. Project managers were testing. It was the norm. And I just started to ask about it.I had a good friend of mine who’s a mom and she had taken a step away from her career to raise her children, but her children were getting older and she was looking for opportunities. I said, “You know, that will be a really cool thing for some of these one-offs, like landing pages and things that we can hand to moms and have them come in and test this.And it’s like what type of requirements do we need? What do we need to share with them? I just had this cool idea of an initiative to like engage mothers to help support quality within our space. And I was talking with the CEO about it and I was also complaining about some quality issues that we had. We were seeing some major quality issues. There was an incident with a cat playing the piano on a landing page. So lots of different issues, but long story short, I guess I talked about it enough that they came to me and they said, “Hey, are you interested in this?”

And I wasn’t at first, because I was setting myself up to be blamed for something. And after a few more conversations and research, I jumped into it and I loved everything about the process, and organization, the collaboration, coordination that was needed to really create a quality department and work with other teams and qualities that we know of everything when it’s done well when it’s in a culture that’s looking for it. And so I loved that and I haven’t looked back like, “developers, the quality, the quality is better.”

Federico:

I don’t want to start a discussion about that.

Erika:

No, no, no, no, no. I love developers and I still tinker on my own, but I found, for me, my passion lied in process, and organization, and structure, and strategy, and that building quality teams and transporting quality teams, for me, allowed me to leverage those passions and the natural strengths that I had in those areas. And I’ve always enjoyed it.

Federico:

Amazing. Perfect. So the main topic today, it’s related to the role of test automation in the whole testing strategy. And to start discussing it, maybe a good starting point will be understanding the test strategy that you have in Calendly.

Erika:

It’s still evolving. When I got there, we had 14 members and they were over-sourced across multiple, more than four squads. And we had test management, but it wasn’t fully utilized. We didn’t have traceability across our requirements with our tests, a number of those things. So getting the foundations in place. 

But our developers deliver automation today, actually. And there are definitely some pros, but there are definitely some cons to that. The cons are really based on the lack of a strategy out the gate. Not out the gate, but when they started some of the foundational structure was not put in place.

And so it’s grown and it’s grown rapidly because the company is growing rapidly. And there are hiccups with that when you don’t have that good foundation in place. So we are looking to change that and we are looking to infuse quality across the business and get QA out of just the review column, get to a way more embedded on the left and the right side of the review column, and the business, getting closer to product planning, and inception, and also in production, what are the things we want to do there from a testing standpoint as well?

Federico:

Yeah, but this is strategy. I think it’s something that typically evolves all the time because you change your focus, you change something in the development process.

Erika:

The company just changes.

Federico:

Yeah.

Erika:

And especially when you get bigger, that’s what we’ve had to deal with. And that’s probably the hardest part is that we’re growing so rapidly. Thank goodness! We’re so excited to do that, of course, but there are challenges when you grow rapidly, that you do have to make changes and you have to sometimes push more process in to wrangle more people, but it happens so quickly that there’s resistance because you went very quickly from this, oh, one pizza team and we can just have a conversation really quickly and get on the same page to we’re a two-plus pizza team and not as quick to have those conversations. And who do you pull in, because you don’t want to waste people’s time? And so trying to infuse that strategy on top of that strategy, a lot of that is documenting and having a strong process, everybody being on the same page to be on the same page. Documentation is a big deal.

That’s the shift that we’ve been making, but luckily culturally within the company, everybody’s really intentional about our core values, and finding a way and striving for excellence, and focusing wisely and starting with humans. So I think that has a lot to do with our ability to scale quickly, and still deliver a high-quality product.

Federico:

You mentioned that the developers do some test automation, right?

Erika:

Yes. Our developers today are delivering our automation. 100% our delivery. We’re working with them right now to make a shift in that area so that they can focus more on units, component-level testing, and then QA can take on more at that middle tier and upper tier testing when thinking of the test pyramid. So integration through exploratory. So we want to make a shift in there, and that comes with talking about different tools and technology, and changing what we’re doing today, like we’re using our stack with a few other testing tools and frameworks.

Federico:

Many times the developers when they also participate in the testing activities, specifically in automation, they rely a lot on that. So they can see the value in the manual testers. Let’s call it manual testers.

Erika:

Yeah. If I’m understanding the direction you want to go…

“I firmly believe that developers need to have a role in testing. In testing strategy, in testing delivery, they need to feel the pain of a test strategy that’s not good. They need to reap the benefits, and be a part of that experience.” 

When it is done well, or when tests aren’t going well, they should be involved in that conversation. Because also it helps to make sure that they understand what needs to be developed off of the app that they’re delivering, that they are developing, and how can they change or improve how they’re developing to improve testing. Right? So it is not a siloed conversation. And when we do that, when it’s just testing, or just dev there’s always problems, we have to make sure that we’re coming together on that.

Federico:

Yeah. Many times this is really important, the collaboration between the developers and testers, and also doing some things in a different way in your development process in order to facilitate the activities of the testers later, or even the automation or whatever. But related to that, something that I’ve been asked many times is, how do you manage the collaboration between the teams in order to be efficient? Because I don’t want to automate something and then have the testers testing the same things that are already automated. So how do you manage the coverage, the strategy around that?

Erika:

So that’s a good point because you can quickly wind up with duplication. We actually went through an audit process of our automation against our test cases and started mapping them and making sure that we’re pulling those into our test management system. So we’re looking at the automation that’s coming in from the developers today, and we’re saying, “All right, what did you build against this new requirement that you’re putting in?” And our testers are saying, “Hey, this looks like it’s a duplicate of this test.” And then having those conversations with the developers, “Hey, do we need to pull this back and put this in this test?” Or, “Is this covered here? Help me understand why we have this brand new test and do I need to create a brand new test case for that, for it to be mapped to?”So we want to continue to have that one-to-one mapping so that… It’s kind of a check and balance, right? This is what testers are expecting is tested. Is it covered by automation? Do we know it’s covered by automation, and then let it report into our test management system so we don’t have to manually test it. So now we’re also reducing our test cycles by keeping track of that. And then we’re able to bring in that as part of our… From a reporting standpoint, we’re able to marry the automation and the manual testing in the test phase of the delivery life cycle.

Federico:

So you have the complete picture of what’s covered.

Erika:

Sure!

Federico:

Perfect. And what about the collaboration with other roles, like product owners or business analysts, or maybe the DevOps engineers?

Erika:

Yeah, yeah. We love DevOps team because they help us with tools. So, that’s really where we’re collaborating with them, is how do we get the speed that we need? What do we need to do to have a stable infrastructure that can impact the stability of our tests? But yeah, for sure. 

We’re talking to customer experience, we’re talking to the product. In our regression cycles, we’re including them as stakeholders as part of those regression cycles. And they’re doing stakeholder testing to make sure that everybody’s on the same page that we’ve said, this is what we expect to deliver and it’s meeting the standard that we want to give out to our customers.

Federico:

Cool. Another question that I think it’s really common is, how soon can you start testing?

Erika:

As soon as somebody thinks about it in their brain! The sooner, the better. But our testers have to get really comfortable with asking why. And that question, or just questioning, and not just saying, “I’m just taking it in. You’re telling me, and I’m just taking it in. This is just for knowledge.” No, this isn’t just for you to gain knowledge. This is for you to critique, you to evaluate, you to question what is going to happen, you to share back information. “Hey, have you considered what’s going to happen if you plug this in here, and what about this particular workflow or edge case, even? What’s going to happen to this, it’s going to orphan this particular user in this flow.” So, yeah, I think as early as… Which we have implementation plans when we’re talking about what do we want to deliver.And quality has a stake in that and getting more engaged. What are the risks? What are the concerns? Even at a high level, you’re not going to have all the answers, but we’ve got to get confident and comfortable with thinking through the idea of a piece of functionality. So it’s that ideation stage where you’re like, “Hmm, I’m going to creatively think about this. What is the purpose of this, and how would I test it?” Because that also feeds into, how are developers going to build it that are trying to come from the backend and say, “Well, now how do I test it,” and have to build extra tools to be able to test it, or they have to go in and muck around with things to open up areas of the application so it can be tested? We don’t want to do that. We want to plan ahead.

Federico:

Asking these types of questions at the beginning before starting with the development, I think is the way we have to prevent errors, instead of only detecting them and reporting them when they are already there. Can you tell me about your main challenges today regarding testing and test automation Calendly?

Erika:

I don’t want you to know, no. We don’t have any challenges. We’re amazing! Right now, automation is our big one. We have a ton of automation and you would think that that would not be a bad thing but we don’t know. We’ve got, our suite is too bloated and we have platens back that we have to get under control that slows our times down. It slows those, our delivery time down. 

But I think we have a good direction. We’re moving in the right direction to solve it and that’s getting the team focused on getting the developers focused on that unit layer, figuring out what that formal strategy is, an expectation, whether it’s lines of code covered or requirement based or whatever we want to decide on that measurement.

Then, building on that to say, “All right, so we want to look at key components and we want to look at integration and then we want to look at contracting and so forth,” and just continue to build on that to say, “Okay, how do we make sure we have this layer of coverage? What’s missing? Now, let’s look at what’s at this layer, how do we fill in the gaps?” Deep bragging. How do we fill in the gaps? Now, go to the next layer. What else is missing? And you just reduce and reduce and reduce, but you know that you have that full coverage.So we’re moving in the right direction. We’re having the conversations and we’re putting a plan in place to get that strategy in that structure. I’m really excited about where we’re going this year.

Federico:

I like that approach of paying attention to the foundation layer and then filling the gaps as a way to direct where you want to automate in the following layer.

Erika:

Yeah, because you say, okay, well what’s left? Now, you’re more intentional. Then also when you’re looking at those layers saying, “Okay, who’s responsible for this at that layer?” And getting clear on that and what is the process and who needs what out of this?From a QA perspective, I’m saying, “I want to know what you’ve tested, because now when I’m building my test plan, I want to account for that. How do I map that to the things that I’m building and the strategy? How do I make sure that it has the complete coverage or to indicate, or to be able to bubble up, hey, there might be something else missing here. I don’t see that we’ve covered this piece and this should be at this layer. Can you add this in?” So you have the checks and balances to make sure that we’re not putting on blinders or just one person’s view of what’s needed.Then moving to that next layer and saying, “Okay, who’s responsible for what at this layer and what is our ideal coverage here? What, what gaps does it leave?” and keep moving up the chain.

Federico:

Maybe this is another way to avoid duplications between the different layers.

Erika:

Yeah, exactly. 

Federico:

Amazing. In one of your talks, you say that manual testing and test automation are not mutually exclusive. I love to delve deeper into it.

Erika:

Yeah, they are not. They are a partnership. They should be working together. I should probably say it more like…

“…Automation works for us, not the other way around. When I say us, I’m talking about testers. It’s not a replacement. It is a tool in our toolbox.” 

We’re automating. We are responsible for telling the machine what is automatable and if we don’t understand it, we are not exercising the system or we’re going off what are the requirements that have played telephone all the way down to delivery. It’s a telephone game. You’ve said, “Hey, did my ones and zeros check out? Did my build work? What is the machine’s interpretation of this?” Then, you’re asking the machine to say, “Is my interpretation, correct?”Well, yeah, the ones and zeros are fine. I’m not saying that automation doesn’t find issues because it does. But it’s those edge cases, it’s that, “I didn’t know you were going to go back”, and then, “I didn’t know you were going to use the back key on this. Why would you do that? Oh, you have a valid reason. I didn’t think about that. It wasn’t in my automation course, so it’s not going to sail gracefully.” You need people who are engaged in the application, who are able to say, “This is what we need to test. This is how this type of customer is going to use the system or need to use the system.” So think about this person’s pain points or their, what is it that they desire to do? What is their end goal?And that end goal, how do I get to that? I might go an unconventional route if I can. I might try to take a shortcut if I can. When you’re not when you don’t have people supporting the automation strategy and when you’re just saying, “Okay, well, we’re separating the two and I’m just going to automate,” meaning I’m putting blinders on. Then, it’s not strategic either, and that’s how you wind up with bloated test suites because you’re not looking across, you’re not figuring out ways that you can defrag your test pyramid, you’re not figuring out how can I assess for risk and be strategic, and reduce the time for my automation suite. Because at the end of the day, you can still wind up with a test that takes you just as long as it does for a manual test.

Federico:

Of course, you do. And similar things apply to the different quality factors like security testing, performance, and other aspects.

Erika:

It’s a planning effort. You plan for it. But to plan it out, you got to, you know, human brain things.

Federico:

Amazing. I have a couple more. I have some more questions different from the main topic. One is how do you improve your quality sense? I mean, this instinct that helps us to find bugs or risks?

Erika:

That’s a great question, actually. I mean, some of it is definitely understanding the application and the business. But I think putting yourself in the customer’s shoes is a lot of it. Like, what would I do? Okay, I’m from a teacher and I’m using this product. What would I do? What would I need it to do? What makes the most sense? I’m trying to think. When I think of those gaps or those questions, what is it that I’m thinking of? A lot of it is experience. It really is.

Federico:

For me, it’s also related to empathy and this is another question I have that I don’t have the answer to yet, which is, how to train or improve your empathy skill?

Erika:

That’s why thinking about is the problem, what is their pain point for that particular persona? Understanding the domain and your customer base, who’s using it, and then thinking about each of them. That’s why I’m pushing to have really strong, clearly defined personas. I think you have sub-personas if you will. Like, this is a teacher’s assistant, this is a principal. It’s education, but there are these different people in this domain and they’re going to use it differently and have different needs and goals and wants and so forth. But empathy, that’s a great, succinct way to say it.

Federico:

I think this contemplates a particular thing related to quality, which is that quality is subjective. You have to think about the people using your products in order to understand the quality of your product. It’s not only what I consider quality.

Erika:

This is so true. I’ve had, people ask me what is quality. I was like, “You tell me.” It’s your definition versus mine, your standards versus mine. You are going to have a higher standard maybe or something else, whereas I might have a higher standard for skating, a skating rink, a roller rink floor. I might have a higher standard than somebody who’s like, “I don’t roller skate.” Well, I do. That’s actually a prime example. I was telling my husband that there’s a bump on the floor over in this section. He was like, “No.” “Yes, there is. There’s a bump. It’s a little one, but it’s there on this skating rink floor.” He was just, you know … experience in standards.

Federico:

Cool. What about books? Do you like to read? Do you have any books to recommend?

Erika:

Well, I’m finding it hard to find a book that I’m recommending other than  these days. It’s my absolute favorite book to recommend for test professionals. I have recommended it ad nauseam lately, it sounds like. But I really do. It’s by Owais Peer and Ronald Cummings-John. Great, great book. Not a long read, not a difficult read, but really talks about how QA should get out of just the review column and that we have more to give. We have more that we can do across the entire delivery cycle. 

Also, find books on influence. 

Influence is a key part of our role in the quality and any book that you’re interested in on influence, just to talk about what does it mean to influence others and influence them in a way that they want to be influenced. 

You know? So that you can talk to people and you can engage them and you can understand them and that empathy so that you can help them come along the journey that you’re also trying to take the lead for quality’s sake.

Federico:

Cool. I’ll add it to my reading list.

Erika:

There are a few books, yeah. There are a few books on influence, but not everybody, I keep telling people different ones. I make no recommendations, not having read them all, but there are a few out there.

Federico:

Amazing. Erika, do you have anything you want to invite our listeners to do?

Erika:

Connect with me. You can find me on Twitter: Erika_Chestnut. That’s my handle. I need to tweet more, but I want one-on-one conversations. You can go to my website at erikachestnut.com and schedule some time. I love to connect and hear about what people are doing in their business. Having conversations like these and helping. I love to coach and mentor others and help them reframe the value of quality and get them in the space of being able to socialize the value of quality and turning the box on how they can infuse quality across their business.

Federico:

Cool. The good thing is that it’s going to be very easy to coordinate a date and time with you.

Erika:

Yes. I use my favorite product out there these days, Calendly. If you have not heard of it, Calendly.com. I urge you, urge you to check it out. It’s amazing.

Federico:

It is amazing to talk with you, Erika. Thank you so much for your time.

Erika:

Thank you so much.

Federico:

Okay, bye.

Previously published at