This month (May 2015) at the Angular Lunch we had a very special guest speaker. Matt Follett (currently working at Observable Networks) shared his extensive experience working with end-to-end testing of browser-based web applications, including Angular applications. Although Matt only briefly mentioned Protractor (the Angular specific wrapper around Selenium), nonetheless almost everything he said was highly relevant to protractor.
Matt did not pull any punches in describing the difficulties of this type of test, and his comments certainly match our experiences at Oasis Digital testing Angular applications using Protractor. Still, the payoff of this level of testing is so high that we find it generally worth the effort. Automated testing is a (the?) way to keep quality up and regressions away as an Angular application grows in complexity and size.
Perhaps the best part of Matt’s talk is toward the end, when he (gently) compares the various available hosted browser/selenium services.
Transcript
We have also transcribed the the talk to text, provided below. This is a rough, first-draft transcription, so any errors are probably in that process, rather than in Matt’s actual content!
Kyle Cordes: I’m Kyle Cordes of Oasis Digital. Welcome to the St. Louis Angular Lunch. Today we have Matt Follett talking about browser-based testing. And I think he was just saying before he started he’s going to disappoint us all with his choice of tools. But that’s okay because I specifically asked him if he would please come tell us about this topic not necessarily from the point of view of Angular and tools, but just from the point of view of having been down this road and suffered extensively doing automated browser testing at scale. Take it away, Matt.
Matt Follett: Thank you. Yes. As he said, my name is Matt Follett. I’m currently a Senior Software Engineer at Observable Networks. Everything else about that side is not interesting.
The important part to think about is I’ve felt this pain. I’ve been doing Selenium tests for a while now. What you end up doing is you write four lines to try and find an element. It doesn’t find the element. You think the selector looks fine, so you try it in a browser and that looks fine. So you add a wait and you pull up DevTools really fast, but that doesn’t work.
So then you put a longer wait and you finally figure out, “Oh.” You thought the ID was camel-case, but it’s snake case. And you just didn’t notice that because, in your mind, you mapped those things the same. Fix the test. You write four more lines. Fails again. Put a really long sleep in. Curse Selenium. Curse Selenium’s authors. Find the issue and fix it. But, of course, you put that long sleep in there, which meant that when you tried to kill things, it actually kept a connection to the database because it kept part of the server lock around. So you just start killing things because some have to die. Start dreaming about living a technology-free life in a cabin by a lake, but you remember you like reddit, and so you keep going.
Push to CI server. Fails again. Change isn’t in the database, so you just put sleeps everywhere, like, somewhere there is a problem. Let’s just just put sleeps everywhere. Push it. Works. It’s the next guy’s problem.
Pack up. Go to a bar.
Things can be better.
Do You Actually Need Selenium?
First thing. And this is an important one. My little example there actually does not fix under this. But the first question you have to ask yourself—do you actually need to be using Selenium for the thing you’re doing right now? There’s a really easy way to know that. You have to ask yourself—is this actually being driven by user input? Is it modifying the state on the server? Are you caring about what the server comes back with? And are you caring about how the user perceives the thing that comes back?
If the answer to all those is not yes, if all you’re worried about is dealing with user input, well, that’s easy. That’s a front-end testing framework. You should just be using Jasmine and not be using Protractor or some Selenium-based thing.
If you’re only caring about what’s happening in the backend, then you certainly shouldn’t be using Selenium because it actually won’t even work that well.
Again, if you’re only caring about how your client responds to the server’s response, that’s not Selenium either.
The only case that you really need to be using Selenium is if you’re actually worried about the user putting something in, it getting sent back to the server, it coming back from the server, and then it doing something. If that’s not the full path that you’re testing, then don’t be doing this. Because you’re making your life harder. In general, fewer Selenium tests, less pain.
You know what? I am actually using the wrong slide deck. This was for a previous one. So let’s pull this one up. I should notice because I made these dark and sad. But I also added, I do believe, this brief note.
One thing that’s important about this talk is that we’re not talking about Protractor. We’re not talking about Capybara. We’re not talking about Selenium, technically. What we’re really talking about is just the process of doing these things. It doesn’t matter which one of these you’re using. Pretty much all of these, we’re going to be wrapping something that’s either like WebDriver or WebDriver-esque. And all of these practices apply across all those places. So I’m not going to say—actually, on occasion, I might say, “In Protractor, you do x.” But that’s only a concrete example of the general things that I’m talking about.
So I’m not saying you should use Protractor, I’m not saying you shouldn’t use Protractor, though, I have my opinions on that. One thing I would say is that if you want to debate me on any of my opinions, I am open for that, but not during the talk because we actually only have 37 minutes left at this point. And I’d like to have time for questions. If you have a question, absolutely interrupt me.
See how dark and sad these are? It’s much better this way. Oh, my gosh. I talk so much.
Anyway, things can be better. Do you really need these things? But if you actually have to do it, remember a couple of things about Selenium. Every test you write, it costs time. It costs time for you to write it. It costs time to maintain it. And actually, literally, costs your company money if you’re using something like Sauce Labs because you’re paying them for the amount of time that you’re running the tests.
There are reasons for this. Selenium tests are slow. They have to be slow. By definition, they’re slow tests because they’re starting at the browser, they’re ending up in the database, and then they’re coming all the way back around again. And they’re doing a bunch of things. And they can be very fragile against race conditions. Sometimes that’s unavoidable. Sometimes it’s avoidable. If you just need to run one Selenium test or something like that, that is a really long process to start up because you’re starting up a web server, you’re probably creating a database, you’re running through a whole bunch of migrations. A single Selenium test has a lot of costs to it.
Not Like Unit Tests
So one thing you really have to remember about this is that Selenium tests aren’t like unit tests, and you shouldn’t treat them like unit tests. You’re not supposed to look at them as, “I want to test a specific single thing on my page.” You want to test whole user paths. You want to say, “All right. What happens when a user fills out the enrollment page but screws up half the fields?” or “What happens when the user fills out the enrollment page but doesn’t mess anything up?” Those are the things that you want to be testing. Go through the actual steps that your users are doing to perform some task. That’s what these tests are actually supposed to be for. If you’re testing anything smaller than that, you’re not really getting the value out of it. One of the nice things about testing these long paths is that all that value of testing all these different conditions that the user might be perceiving helps amortize the cost of having all that startup problems and all the different little things that come with any given singular Selenium test.
But I call this “Don’t have to suck”. I probably should say it doesn’t have to suck as much. A lot of the problems you’re going to have, they’re going to exist.
WebDriver REPL
One quick thing that you can do—and Protractor has this, so congratulations if you’re using Protractor. I don’t actually know of another framework that has this, but they’re really easy to write—is if you don’t have one available from your framework, write yourself a REPL for this. Something that you can start up and it’ll bring up a Chrome instance or whatever browser you want to use, and it will actually allow you to execute commands against Selenium to drive that page.
That is a huge savings for writing Selenium tests. Because, instead of going through the process of writing four lines and then seeing if it worked, and then writing four lines and running through the whole test again to see if it worked, you can actually write it one line at a time: Find button by text, submit, and see if that works. Oh, about a half second turnaround, it found it. Okay. Now, click that. Okay. And now, you can keep going from that. So you actually get the rapid development that you’re used to in other cases.
Yes.
Speaker from the audience: [unintelligible 8:59]
Matt: Go for it.
Speaker from the audience: [unintelligible 9:02]
Matt: No, I haven’t. No, I’d be curious about that one. Oh. Yes. The question was, “Have I played with Elementor?” I have not played with Elementor.
Speaker from the audience: It supposedly stands for element explorer [unintelligible 9:32]
Matt: Huh. Apparently, it stands for element explorer. This paradigm thing is going to be interesting, I think.
Bonus REPL
There’s a bonus. Some libraries—. Interestingly enough, Protractor or Capybara, if you happen to get it poltergeist working, which I never did and I don’t know anybody has, but if you get this thing working, you can actually set the debugger and you can start a debugger inside of a test. You can say, “If this fails, pop up the debugger. And then I can actually explore the state of my application at that point or the state of my web interface at that point.” And then you can actually go and start looking as to why is it that it failed. Was it false assumption? Was the test actually finding something wrong? Or was it that I dorked a locator?
Speaker from the audience: [unintelligible 10:26]
Matt: That’s a good point. If your language supports some sort of “Drop me into a REPL”, you can actually just use that. You won’t necessarily get a REPL and—if you’re from a headless mode, you won’t get pushed into a headed mode or something like that. But if you’re already running it in Chrome or something like that, you can drop into a REPL and then that will pause the test and you can do whatever you want. That’s a very good point.
Handling Change
One of the things I mentioned is that writing browser tests, they’re often very fragile to change. There are a couple of things that you can do to try and mitigate that issue.
Finding Elements
Your fragility is going to come from finding elements. This logic can change a lot because the page is going to change a lot. And when the page changes, that’s going to change how you find things on that page.
There are two ways that you can solve this. One is you can extract that code out. Try and make it easy to find, easy to change, easy to have as few side effects in changing that as you want. And you can also try and just make sure that you’re using the best selectors.
Use Locators (If Available)
If you have locators which—it’s really weird that locators is the term that’s used—then you should absolutely be using those. The actual Selenium library has them. Python Selenium library has them. Protractor has them. WWW::Selenium and Perl, which I don’t think anybody in this room is using, has them but they’re kind of weird.
Speaker from the audience: What is it—locator?
Matt: I will get to that right now. A locator’s actually really nice. Here’s an example of a locator in Protractor. A lot of times, you’ll say, “Click on the element that has this button text or something like that.” What you can say instead, is you can actually just make a locator that’s just singularly the “how I find this thing”. And if you take these and you put them at the top of the class that you’re building, the top of the file that you’re working on, they’re easier to find. They’re easier to modify. You’re keeping these in one spot, so you’re not having to change it everywhere when you change the page.
Even if your specific framework doesn’t explicitly have locators, it’s really easy to make something that’s a lot like a locator.
Speaker from the audience: Is there a return in that thing, or is it really just that statement?
Matt: Oh, yes, there should be a return.
Speaker from the audience: Okay.
Matt: Yes. This code is not necessarily correct. It’s an example. There should be a return, but if I put a return there, that would have been too long.
Speaker from the audience: [unintelligible 13:31]
Matt: Style. Say what?
Speaker from the audience: But incorrect.
Matt: But incorrect. Whatever.
Benefits
Benefits of this is that, like I said, it’s easier to find these than it is to find all of the little places that you’ve done something. You put it in one spot. You don’t have to change it everywhere. It’s much more dry. It’s separating the how you find it vs. what you’re actually doing with it.
Use the best selector
The next thing is you want to use good selectors. What the internet tends to agree with is that the best selector to use is an ID. Then you go down the list, all the way down to XPath.
I actually don’t agree with this. And I think, by ID, when you’re doing Angular stuff is actually really dangerous because you really shouldn’t be using a lot of IDs in your directives. Because anything that you can have multiple times on a page shouldn’t have an ID attached to it. So ID very quickly falls off.
The other thing I don’t like about IDs is the user doesn’t know what the idea of any of your elements are. The user doesn’t know the name of any of your elements. The user doesn’t know the class of any of your elements. So if you say, ID foo has this value, or click the button with ID foo, that’s not helpful. Because if the text of that button isn’t Submit, or something that indicates what the button is doing, then the users are not going to click when they actually get there.
My big preference is actually this. If you can, go with the text. That’s what the users are actually seeing. That’s what they’re actually going to interact with. Some frameworks actually make it really easy to use labels for fields on a form. If your framework does do that, then use that functionality with a caveat that if that framework is Capybara, it’s a little inconsistent in how often it works. That one’s actually always consistent that if it worked the first time, it will work again. It just might not work the first time.
So my preference is use text, if possible. And then, honestly, Name, Class, ID, those all fall under CSS selectors, anyway. Just [use these as 15:56] selectors because they’re going to be what you—. Actually, you could use CSS selectors for partial text also because [unintelligible 16:03] contains—. But my biggest preference is text. Also, text doesn’t actually change all that much on a page. It really is hard to get people to agree on how you should change the lingo that you’re using that prompt the user. People argue about that really vociferously. So it’s easy to trust that that’s not going to change.
And don’t use XPath. If you have a lot of familiarity with XPath and you need it, then go ahead. But the problem is, is that if you’re wrong with your thought that you have a lot of familiarity, it’s going to work great until somebody changes the page in a way that you didn’t know about and your XPath didn’t handle properly. And now, you’re falsely selecting something else altogether. Avoid XPath. Just don’t use it. Plus, frankly, you can probably get away with CSS selectors, anyway.
Page Objects
Page objects—this is the best way to manage dealing with the fact that pages change. If you’re not using page objects, the one thing you need to walk out of here with is to start using page objects. What you’re actually doing is you’re defining class that actually defines how you interact with the page. So in your tests, you are no longer saying, “Find this ID, click on this thing, find this thing, check the value.” What you’re actually doing is you’re using this page object to say, “Fill out my form” or “Click the Submit button.” Whatever it is. That sort of thing.
What that means is that all of your tests that are interacting with this page, now they’re interacting with this page object. And now, when you change the page, you only change the page object. You don’t have to change all those tests. You’re not supposed to be doing any sort of assertions with page objects. But it allows you to put all of your selectors in the page, all of the interaction with the page, all the workflows you might use. Eventually, when it get to the level where you have on your page objects the “Fill out my whole form. I’m going to give you a hash of key value pairs, and you’re going to find all the inputs, and fill them all out properly.” And I don’t even have to worry about that anymore.
The traditional definition of a page object is that, when you’re transitioning between pages, the page object actually references the new page that you’re transitioning to. That’s probably a good idea. I’ve seen that violated without any pain or consequences but either way.
Page Object Example
This is a really tight example of a page object, which almost certainly has errors in it. So don’t copy and paste this into your code. But basically, the idea is, we’ve defined all of our selectors up top, so now we have an easy selector for, like, firstNameInput, lastNameInput, usernameInput, passwordInput.
Tiny thing, side note about Protractor is that Protractor is actually lazy about these things, so we’re defining these here, but it’s not actually finding them on the page, which is good because we’re defining them in this function before we even get to it. But you can safely, in Protractor, use a locator on a thing that doesn’t exist, and it won’t try and see if it exists until you actually try and use it to locate something.
But then we have functionality in here. So we say, “populateForm is this thing that’s just going to go through and set all the fields properly.” I had a whole bunch more examples, but I had to delete them all because they wouldn’t fit on this slide. But you get the idea. This one page object that deals with the Registration page—it deals with all the interaction with the Registration page. It deals with [unintelligible 20:08] everything on the Registration page. In fact, it actually even deals with going to the Registration page. You create a RegistrationPage and you say, “Get it.” And it’ll go to that page. I always name my methods goto. I don’t know why I named it get in this case. You get the idea.
That actually is violating the Page Object Pattern technically because you wouldn’t be getting the page and then going to it. You would already be at the page because something would have returned that for you.
Page != Page
It’s also an important thing to note that the Page Object Pattern is actually kind of interesting in that it doesn’t have to be a page. I very frequently use this Pattern for just complex elements on a page. So your page object can consist of a whole bunch of basically directive objects—or something like that—that are actually managing the complex manipulations of inputs and user interactions on little aspects of the page. And that simplifies that. So now when you change the directive, you change the object that represents that directive in your tests, and you don’t have to change it everywhere. You just change it in that one spot, which is really useful in Angular because that directive could appear on a whole bunch of different pages and you don’t have to go through and change all the corresponding page objects for that.
The pattern is called Page Object Pattern. I actually use for lots and lots of things. Of course, this [unintelligible 21:39] apply to Angular [unintelligible 21:39] with Brick or Ember, Polymer. But I have no idea why you would do that.
UI Design
Another thing that seems to be a common pattern—and actually some communities are a real fan of this and I loath it, is—. There’s some aspects of UI design that just, frankly, make testing easier and make usability easier. Don’t lie to your users and say something was done before you even started it. Because when you do that, your Selenium test is going know. Your user might not ever figure it out, but your Selenium test will—about 30% of the time—and that’s the random failures you’re getting on your continuous integration server. So don’t tell a user a task was done until it was done. Because if you tell it beforehand, Selenium will believe you and then it’ll call you out on it when it goes to check the database if your test is actually doing that, to make sure that the thing actually saved. Or when it brings up another page or something.
And also, loading indicators—they’re great for users because then the users know something is going on. And they’re great for your Selenium tests because you can hang weights off of those indicators. You can say, “Okay, wait for this loading indicator to pop up. Within that context, extend the amount of time that I’m willing to wait because I know something’s going on. Once that loading indicator goes away, I can set my wait back down to a much more reasonable time and go from there.”
You have to be a little bit careful with that because if you’re using a smart loading indicator that waits a little bit and then doesn’t show up if it happened really fast, then you have to be conditional with your waits and that sort of thing.
So [it’s logic 23:36], it’s a little bit more complicated but I really recommend these things. I think not lying to your users and letting them know what you’re doing is generally a good thing. And it turns out it’s helpful in testing.
Getting on the Grid
So let’s say, you’re actually pretty happy with your tests and things are going well. Let’s talk about, actually, cross-browser testing and using a service.
First off, you’re probably not going to set up your own Selenium grid. But you’re probably going to use a service that exists for it. Your options are BrowserStack, Sauce Labs, TestingBot, a bunch of other ones that I’ve forgotten while I’m standing here. They all suck. They’re all really bad. I mean, they’re great for what they’re doing, but they’re all slow. They’re all going be very slow. And, by very slow, I mean, whatever your CI server runs your tests at, multiply that by an order of magnitude and that’s a rough approximation of what’s it going to take. And to some extent, that’s not their fault. There’s a whole internet between you and them. But they’re also running on the cheapest VMs they can get out of Amazon, so some of it is their fault. Some of it’s the internet, some of it’s they’re saving money. It’s a company. I can’t blame them for that. It’s a business model. Yes.
But when you’re looking into these, there are a couple of features that you certainly want to have that not all of them provide. If you can get videos—I know Sauce Labs does videos, I think. I think BrowserStack and TestingBot do. If you can get videos of the things, that’s actually really nice because when it fails, and it will, you can actually go back and play through the video and see what happened.
The alternative to that is most of them will give you snapshots. But the snapshot is just the point where it failed. Plenty of times, the snapshot is a page that actually didn’t load or a page that you didn’t even expect to be at. And so, how you ended up there is actually really important. It would have been nice if you had a video. Or in some cases, you haven’t been cross-browser testing before, and now you’re actually testing on an old version of IE that you claimed you were supporting, and the page is so incredibly broken you don’t know what you’re looking at, anyway.
They also provide logs. But the logs are generally useless. If you have to rely on the logs, good luck with that. I have no advice for you except don’t rely on the logs. Find some screen shots or some videos.
Your Test Will Fail
Once you go on one of these services, your tests are going to fail. And it’s going to be because you’re not expecting how slow these tests are going to run. So you need to prepare for this.
In the past, the way that I prepared for this is, I just accepted it. And then had the headache of trying to figure out how a test could have failed, by either watching the video or just playing it through my mind. It’s, like, “Okay, why would it have failed there? What would it have to have been slow for that? What would it have to have not responding correctly?”
Proxy for Slowness
There’s actually a much better way to do this which is that you can actually just use a proxy server for this. I actually say Charles Proxy here is Mac only. That is untrue. Charles Proxy runs on everything. It’s actually just a Java app. But I didn’t realize that. So, sorry.
You can actually set up a proxy in-between the client and your actual server, and just enforce slowness there. You can also—. Because Selenium uses the WebDriver wire protocol, I think it’s something like that, and that is an HTTP protocol also. So if you want to be really mean to yourself but preparing for these things, you can put a proxy in-between both your client and your server, and your tests and your Selenium instance. And then slow them down. You’ll actually recreate the conditions that you’ll experience on one of these services, except in the comfort of your own home. And you can actually watch it and see why it’s failing. That is super useful. I highly recommend that.
Charles Proxy, if you haven’t looked at it, it has a nice GUI. If you’re not comfortable with command-line stuff, then I would recommend that one. BrowserMob is harder to use if you’re not familiar with doing stuff by the command-line and that sort of thing. And also, I think, the docs are—not really is much there. But you can use that. I would recommend going with whichever one you want. Obviously if you’re going to try and automate this process, which is an interesting idea, you would want to use BrowserMob. And there are probably others. Those are two good examples.
Picking a Service
If you’re actually going to go through the process, I mentioned a couple of these. The ones I’d recommend considering—TestingBot is really cheap but my experience has been that it’s not nearly as polished. But like I said, it’s really cheap. If you can get over the fact that the docs are not quite as nice as other ones, then you’ll get by just fine.
Sauce Labs is the pricey alternative. It’s honestly not any faster. I really hoped that one of them would compete in that space, but neither seem to. The docs are better. To be honest, if you read the docs on Sauce Labs and then go and do stuff on TestingBot, that probably works fine. But I’m probably going to be hated for having said that.
Speaker from the audience: I missed a bit. You’re talking about the slowness problem in the proxies? You tried to simulate it?
Matt: Yes.
Speaker from the audience: I’ve seen slowness where a test fails. And I can’t tell if it timed out because it broke or because it just hasn’t finished yet. And so you keep turning up your knob on how long you’re going to wait? So then you’re paying one these companies a lot of money? But you still don’t know if it’s broken or if it hasn’t finished yet? Any thoughts on that?
Matt: If you go with the proxy thing, you can actually, basically just simulate that whole process there. That’s one approach. It really gets down to—you have to sit down and either simulate it locally or you just have to think long and hard about it. Yes, it really kills me to think that the solution to the slow services is to use more time, which is what you’re paying for. Yes, it’s a great business model. I think I’m in the wrong business. No, I don’t have a better answer to that.
The third option I’d recommend looking up, by the way, is BrowserStack. They actually have an interesting pricing model. It’s a lot nicer. But the weird thing is, is that they separate the cost of automated tests vs. manual tests, and you certainly want to have manual tests.
If you run into a case where, like, “Oh, this is failing and it’s failing on IE9. And I don’t know if it’s failing on IE9 because it’s actually a legitimate issue, or because it’s an issue with the test, or it’s an issue with the service.” Then at the very least, if you’re using a service that provides you the ability to manually test it, you can bring it up on a VM that they’ll provide you. And you can run through the test manually and see what happens.
That’s often the extra variable in there. It’s, like, “Oh, it failed on IE9. Was it really IE9, or was it Sauce Labs, or was it me?” You just have to start eliminating those variables. Having the ability to bring up a VM and manually test things, which Sauce Labs gives you for free and TestingBot recently started giving you for free, is really nice. BrowserStack gives it to you and if you add the prices together, it’s actually still a very reasonable price. But then you have to get your boss to agree to pay for two things instead of one thing. I don’t know what your business organization is like because that might be a problem.
Questions?
Cool. Any questions? Because we’re at about 12:50, so I figured I’d give myself 10 minutes to defend my state on anything that anybody disagreed with, through questions implying disagreement.
Speaker from the Audience: [unintelligible 32:14]
Matt: Yes. What he was saying was that some of these services aren’t even running Windows on VMs that meet the required resources available for the version of Windows that it’s running. And that, within itself, can cause problems. Yes, the vigilance is the best answer to that. There’s not a better one.
Any other questions? Yes.
Speaker from the Audience: I suppose some people might choose to focus more on unit test or component level test. They do a bunch of things with Karma. They might even go to an extreme to avoid this sort of pain that you talked about and only do that. It doesn’t seem good. Seems like you want to have some end-to-end test in addition to those. But finding a balance there, I guess, is an interesting thing to achieve for. In your UI testing, were you doing a lot of that kind of testing in addition to this?
Matt: Yes. My rule of thumb—and I’m not going to take any questions from my old team for the remainder of this, just in case they imply otherwise. But my rule of thumb was that if we were implementing a feature, it had to have at least one feature test around it. It wasn’t a rule of thumb I had until later on. So there might be some cases where it’s not the case.
But you certainly want to have at least one feature test. You probably want to have a few that cover the weird different cases that users are going to go through in this whole thing. But what you want is at least one of those, and then you want a whole lot of unit testing around that to verify all the other cases. It’s not the case that you need to test every possible variation of how a directive is going to manage things. What you’re really testing with this is how the users are going to use it and how they’re going to go through this thing.
It should be like a pyramid-type thing. The number of user acceptance tests you have or Selenium tests you have should be a lot smaller than the number of unit tests that you have. But you should certainly still have Selenium tests. If you don’t have those, then the only person that’s actually testing that all your things go together is your user. Failing to properly test your UI is the one case where you guarantee that the users are going to catch it. Because that’s what’s the “U” stands for. You can have failures in a lot of points of your system, but the user may never notice. But they’ll notice when the UI is broken. And they’ll probably trust your team less for that than they should for things that are probably more concerning. [Sorry, I violated my rule. 35:39]
Speaker from the audience: [unintelligible 35:40]
Matt: Okay. What’s up?
Speaker from the audience: [unintelligible 35:42]
Matt: Yes. That’s a good point. Sauce Labs does support running your unit tests in different browsers so you can make sure that your unit tests actually pass in all the crazy browsers that you support which is actually really helpful. I think that there is a way to get those to work in other ones. But I have not personally done it yet. Most of these support some amount of mobile. But mobile minutes are always kind of funny because, like, if you’re testing for iOS, that means you’re on a Mac, which means that they’re probably billing you differently than they’re going to bill you for everything else. Plus they can because that’s a really popular thing to be testing right now. So yes. Of the three that I mentioned as ones I recommend, all support mobile testing. But they count those as different minutes, I think, for the most part. I could be wrong. I don’t honestly remember.
Speaker from the audience: It seems like caveats [crosstalk]
Matt: Yes. More caveats. Probably more. Anything else?
Speaker from the audience: I have two questions.
Matt: OK.
Speaker from the audience: [unintelligible question 37:01] Second is, if the writers give you this huge puke of a list of browsers and versions, if you’re trying to support just most common browsers, how many different scenarios would you be testing? Three? Or an old and a new on the top five for a total of 10? Every combination for a total of 175?
Matt: Yes. First question—how hard is it to set up your own Selenium grid? I’m never actually done it, so that is the caveat here. I’ve always just paid somebody else to handle that. But I would suspect that the hardest part of it, I mean, A. If you’re planning to do that, I’m assuming you’re of a size where you have somebody already managing operations-type stuff, systems-type stuff. In which case, it shouldn’t be any harder for them to have a Windows 9 VM for that, then a Windows 9 VM for something else, I think. The thing that is actually more daunting in my mind than managing that sort of thing is trying to find a Windows XP license to be able to test on that. Finding licenses that you aren’t able to just casually come across anymore is the hard part for getting up to speed on that.
The second question is determining what browsers and operating systems you want to go with. Frankly, I’m not saying this is a good idea, but my general take on testing against IE was I would always just go with latest version of Windows that supported that particular IE. And then because I trust IE less than I trust other browsers, I would say that if we said we supported a version of IE, we had tests running on that version of IE. For Chrome and Firefox, I trust them more, so it was a reasonably recent version of Chrome or Firefox. And I think if you’re using one of these where they charge you different for Mac minutes vs. Windows minutes, then you throw the Mac minutes at either Chrome of Firefox and trust it. It’s probably about the same.
Speaker from the audience: Do you test it on Safari?
Matt: I have never worked someplace that officially supported Safari. No, that’s not true. I now work at a place that officially supports Safari. But no, not yet. Safari has some crazy weird bugs, so you should probably do that.
Speaker from the audience: [unintelligible 40:05]
Matt: I always advocate for having an explicit list of “these are the browsers that we support”. Like a policy run. We support the n latest versions of IE. We support the n latest versions of Chrome. Which realistically, that’s an overly zealous policy, but I think it’s a good one to have. And then those are what you test against. Except again, I trust Chrome and Firefox to not be crazy, so I don’t necessarily test against all of them because that’s a lot of money.
Any other questions? Cool. Well, thanks. It is 12:59. Awesome.