One of the most common jobs people ask me about is being an SDET (Aka. Tester). People are afraid that being a tester means that you never get to code, you are isolated from the “action”, and that it's boring. 

My other favorite question that I get when I offer people tester positions is: “What do I do while the machines are testing? Am I free to go or can I play video games while the testing is in progress?” That couldn’t be farther from the truth!

To dispel some of the myths surrounding this job, I would like to ask a few questions to an actual tester, Justin Marks. Justin was most recently a Lead SDET II for the Windows Shell team, but has just switched to working as a Program Manager II.

The Windows Shell team was responsible for the outer layer of Windows during the Vista product cycle.  This includes key features such as the Start menu, Windows Explorer, the control panel, and any other feature that sits on top of the core Windows APIs.  In addition to the UI, the shell team provides a broad selection of public APIs for accessing the Shell. 

Here is my interview with Justin: 

What brought you to Microsoft?

After living in Miami, Florida for the first 20 years of my life, I decided to go to a school with a “better” climate so I matriculated to MIT.  During my freshman year, I applied for an internship as a SDET at Microsoft and was rejected.  Staying persistent, I applied again the following year and was accepted into a (Systems Engineer) SE position.  SE is a very unique position that mostly deals with configuring servers (basically, IT management on steroids). 

I came out to Seattle for the summer and worked for the Office Update team.  Over the summer, I spent a lot of my free time playing racquetball, eating at a ridiculous number of Teriyaki restaurants, and completing my MCSE certification exams (I was totally into IT management back then). 

Before the internship, I never intended to work at Microsoft as a full time employee, I just thought that it would be a good place to spend the summer and gain some experience for my resume.  Fortunately, I absolutely fell in love with Seattle and Microsoft.  The Northwest isn’t Great for nothing!  By the end of the summer, I had interviewed for a full time position, been accepted, and decided to graduate MIT a full year early just so I could get out here faster. 

When I came back to Microsoft, I worked for a year and a half in MSN running the day-to-day operations of MSN.com and then decided that I wanted a position where I could be more technical and work on the development of a “box” product instead of a service.  Hence, I became a tester on the Windows Shell team.

What is an average day like for a tester at Microsoft?

It’s hard to describe an average day for a tester, simply because no day is average.  Not to demean the other disciplines, but the tester is really the one doing all the juggling. 

Common tasks of a tester include: reviewing specs and dev designs, writing test plans, coding automated test cases, running manual test cases for things that can’t be automated (God forbid!), filing bugs, reviewing SDK documentation, “Self-Hosting” (a term used around Microsoft that simply means using the latest build of the product on a daily basis and looking for bugs in and more importantly, outside your feature areas, filing bugs, reviewing dev code changes, reviewing “resolved” bugs and activating them again when the dev didn’t really fix the problem), filing bugs, working with real customers on issues, attending trainings on a variety of topics, reading, triaging bugs with the feature team, representing bugs in triage, war, etc.  

I know that this is a big list, but you really do all of this every week, if not every day.  For me, my goal as a tester is to be the last line of defense for the customer. I have to ensure that the customer will be happy with how the product is, not some dream of what it should be.

How much coding do you do?

Again, it depends (I’m starting to sound like an accounting class).  In general, testers write as much code as possible (why manually test when you can write an automation to do it for you).  In addition to writing automated tests and libraries of functions to support them, many testers are involved in side projects for tools to solve different problems that the organization is facing. 

A great example of this was a tool one of my peers worked on which enabled manual testing of common controls.  The tool allows a user to setup a control (like a button or progress bar) with any settings they wish and then interact with it either manually or by APIs.  This dramatically improved manual testing for the area as well as reduced the amount of time needed for creating repro applications – just run the tool and do what you want, don’t waste time writing something custom each time a new scenario has a bug. 

Today, this tool is used by the entire feature team and has gotten accolades across the org for being a HUGE improvement in the way the team designs, tests, and develops new features.

What has been your most rewarding moment as a tester?

My most rewarding moment as a tester has been getting to develop the model for performance testing across my entire team.  I was tasked to take on the “performance project” after 3 other testers failed to make a large impact.  Working with one of my colleagues, we were able to create a complete end-to-end infrastructure (by pushing “go”, machines would install the latest build, configure the OS with files, programs, etc. and run a collection of performance tests, automatically analyzing the results and filing bugs). 

The project was so successful that we are expanding it for Windows 7 to a broader audience across the Windows client instead of just the shell team.  The project gave me the opportunity to not only design and code the infrastructure, but to broaden my technical knowledge of the entire Windows project.  Performance issues are not contained to individual feature areas – often they span many features and teams. 

During this project, I worked with members of the entire Shell team (300 people) on a daily basis and really got a feel of how the breadth of the OS fits together instead of looking at things from a depth based perspective, as is the case when working on just a single or couple of feature areas.

What are peoples’ conceptions of what a tester does?

Outside of Microsoft, people think of testers as the ones doing grunt work, manually installing products, and testing them haphazardly by just poking at them.  The reality is that testing is much more thought out and methodical. 

With a product the size of Office or Windows, it’s important to think about what test cases matter, how features will interact, and how testing can be reproducible. We want to understand the bugs so we can fix them, not just report that something went wrong, but we don’t know where or why. 

Inside Microsoft, testing is an equal discipline to Dev and PM.  They are not the “wicked stepbrother” that people outside the company often assume them to be.  Unfortunately, to the testing “profession”, there are very few resources on how to test software well.  In addition, most colleges barely touch on the subject: “don’t forget to test your code” is about as far as software design classes go. 

It’s my hope that as the software industry continues to mature (don’t forget, we’re all pretty new at this), a bigger emphasis will be placed on testing as a whole. Some best practices from Microsoft, as well as many other companies in the field, will be shared and taught to individuals interested in software design.

What skills are required for testing?

Generally, if you ask someone what skills are required for being a tester, you often hear things like, “be good at breaking things”, “be inquisitive”, or “take apart toasters”.  While these are all good skills, the reality is that a tester is no different than a developer.  There’s a reason why the title at Microsoft is not Tester, it’s Software Developer in Test.  WE’RE DEVELOPERS!  We just happen to develop tests and tools, not the product itself. 

When I interview SDET candidates, I look for intelligence and coding skills first, and testing skills second.  If you can code well, you know how to debug code and fix the issues in it.  That’s what we do, just on a broader scale.  It’s nice if you can think about a list of test cases.  Try it on your next software problem set – how would you test the function you just wrote?  What would a good list of test cases be?  The list should be LONG, like more than 50 cases (and I’m not talking about putting the numbers 1-50 in as inputs). 

Think about things like globalization: what happens if your function was being run on a system in Germany? What about in Israel, with right-to-left languages? What about accessibility: if someone is blind can they use your program? What about border cases? What about security?  Testing is not easy, but if you’re methodical, you can usually come up with a lot of “creative” test cases.

What career path is available for testers?

There’s no difference between a tester’s career path and a developer's.  Do you want to go down the path of management and become a lead or manager of managers? Do you want to become more technical and move into a test architect position? Do you want to own more feature areas?  Do you want to own horizontal areas like performance or security?  Do you want to become a Dev or PM later on?  You are in no way limited to career choices just because you’re a tester. 

Thanks, Justin!

- Janelle