Welcome to the High Volume Hiring podcast. I'm Steven Rothberg, the founder of Job Search, site College recruiter. We believe that every student in recent grad deserves a great career. This podcast features news tips, case studies, and interviews with the world's leading experts about the good, the bad, ammy ugly when it comes to high volume hiring. Thanks for joining us. Today's guest is Josh Milit, the founder and c e o of Criteria Corp. They're a leading provider of pre-employment assessment software. Josh, welcome to the show.
Thanks for having me, Steven. Nice to be here.
So for listeners who might not know much about you personally, or what criteria does how about if you take a minute and tell us who you are and what you do?
Yes. So I co-founded criteria about 15 years ago. As you mentioned in the intro. We're a pre-employment assessment provider. So our software helps companies make better hiring decisions, better talent decisions. We are active in most of our employees actually are in the US and Australia. We have customers around the world, about 4,500 companies now use our, our software regularly to help manage their hiring process. And yeah, excited to speak with you about grad grad hiring today.
Awesome. And and speaking about grad hiring, just before we got on you shared with us that one of your close friends is an alum of the University of Minnesota, where I graduated from the best school in the world. I think we can all agree on that and and that he has shipped more than one university of Minnesota Golden Gopher outfit for, for your kids.
Yes. There's a steady stream of, of gopher paraphernalia that, that comes in. So
Shirts, they, they certainly are well outfitted my, my two boys in in gopher gear. So
<Laugh>, well, noth nothing says stylish like maroon and gold. So <laugh> and yeah, one of, one of the, well, we'll get into it definitely as, as the conversation progresses, but Josh and I also sh share coincidentally a a we're both Canadian citizens by birth and also American citizens. So we, I think it's safe to say that we both have kind of a global outlook, and we realize that where he's at in California, where I'm at in Minnesota, obviously both in the us we understand this isn't the way it is all over the world. And, and that's one of the reasons I was really excited to talk to you. So I guess the, the first question that I was hoping we can dive into is that the, the biggest challenges that you're seeing for employers with graduate recruiting, that the, when employers are hiring s what in the US we call seniors and some other countries, fourth years and some other countries, freshers, but what are the biggest challenges with hiring those people?
Yeah, excellent. So, so besides the, just the, the issue of very high volumes that are, that are seasonal, right? So that grad hiring tends to happen in a one or two month period. We, we do active grad hiring with our customers in Australia as well as the us and it tends to be very seasonal. So all I know that's the focus of your podcast. And so all the issues that you have with higher that with high volume hiring of, or of course there with grad recruiting, but in addition, you kind of get a new set of issues as well. And I would say the primary problem is that the traditional tools that companies typically would use to help make selection choices, help make hiring decisions and namely the, the big one is resumes, right? They don't work all that well in normal situations.
And they really break down, I think, in terms of their utility in in grad hiring, because obviously these are early career or beginning of career people and their resumes are often basically empty, basically blank, right? So they may have intern experience, they may even have a little work experience, but the typical grad hire that a company looks at will have very little experience on their resume and in fact very little of anything on their resume. And so that tool that's so central to the hiring process really becomes fairly useless in a, in a grad hiring scenario, right? You get, you get a little bit of a sense of their educational record and there's typically not much experience. And as we know, experience is pretty central to how employers typically or traditionally evaluate. So you really focus on measuring potential and the existing tools for, for doing evaluation of, of candidates just don't, don't work very well in terms of being forward looking, right?
They're all oriented around being backward looking, what have you done in the past? You know, where did you work? What industries have you got experience in? Those just aren't relevant in, in grad hiring often. So you need other tools, other kinds of signals to evaluate people in an efficient way. And, and by the way, I think also the, the fact that we still use traditional tools a lot in grad hiring, a lot of, a lot of companies still do is one reason that the unemployment rate for recent college grads is historically in the US being two to three x that of the rest of the labor market, you know, so, so the system's not really working well for either side there for applicants who don't have anything on their resume and for employers who are, who are essentially staring at blank resumes.
Yeah, no, that's well said. It's, if I can build on that too, I, I see in the marketplace and have for years so-called matching technology where you take a, the software takes a look at the job description, which is a forward looking document. This is what you're going to be doing. And it also takes a look at the resume, which is a backward looking document. This is what you've done, and it tries to match that. Well, somebody like you or me who have been in the workforce for decades, it's kind of easy to do that. You know, if you've been an accountant for 30 years, that's a pretty good chance that that job posting for an accountant position is gonna be a good match. And that the physician posting not so much. But yeah, like you say, it's like for, for, for decades and decades, employers have looked and they've said, oh, okay, you have a business degree, you have a marketing degree, you have an education degree.
Well, then you must be qualified for this role. Not necessarily. We have at college recruiter, we've had employers that have said with pride that after two or three years, 20 to 30% of their hires are still with them. Well, to me, that's about an 80, 70 to 80% fail rate. And, and I think if, you know, if they, I think if they did a better job of matching them as assessing them based on their, their, their, their skills, their capabilities, how productive they're likely to be maybe they could get that down to 30% turnover rather than 30% retention. So Josh, the, you did a great job of outlining the, sort of the challenges for employers hiring new grads. How do those differ for employers who aren't engaged in, in graduate hire? They're just, they're, they're, they have roles to fill. Maybe some of them are students, some of them are recent grads, some of them are people who have been out for 10 years. How does that look differently from, from the perspective of, of of doing assessments?
Yeah, I think it's an order of magnitude difference, not really a a qualitative difference. I mean, our view at criteria, and, and I think this is certainly backed by research, is that resumes even in the best of conditions are, are full of what we would call pretty weak signals. You know, so things like educational pedigree, the amount of years of experience you have in a given field, they have some predictive power in terms of, you know giving you an indication of how successful someone might be in a role. But they are what, what statisticians would call pretty weak signals. They're only loosely correlated with outcomes. And yet they dominate how we evaluate people, you know, experience in particular, which is really, you pointed out a resume is fundamentally a backward looking thing, right? It tells you what, what this person has done in the past in terms of occupational history you know, years of experience in different roles, and then gives you some sign of educational background.
And both those things, unfortunately, are pretty weak sigmas. So, you know, what do you do in that, in that scenario? I mentioned in grad hiring the, the weak signals break down altogether, right? Because there's almost no signal there other than educational pedigree. You, you don't have any work history there often. But even in normal scenarios, you know, we, our, our customers will look to our assessments to give them a forward looking indication, you know, a signal around. And there's a whole variety of different types of assessments that we can get into to what the different types of assessments do. But, you know, when you look at things like problem solving ability or the ability to learn and digest and apply new information or work ethic, you know, those are very reliable research shows as a signal of what someone may be able to do in the future, how they might be able to evolve and grow in the future.
And they are fundamentally forward looking signals. So, so, you know, I, I don't want to get too far down the rabbit hole on resumes. We, we have a very strong view there. We don't expect that employers will stop using them. They, they have a purpose. But they are really you know, you, you talked about some of the bad results in grad hiring. The statistic that I see everywhere from a it's being echoed in a, a series of studies is that the hiring success rate in the US sort of wri large is only 54%. It's kind of slightly better than a coin flip, right? I think we can do better if we start using tools more carefully,
54% hiring success. How do you define that? Is that somebody who's stays with you for a year for five years? What, what, where, what, what's that measuring?
Yeah, fair question. I think, I think it was like a 12 to 18 month view. And obviously, you know, the definition of hiring success will really differ by industry, right? In a high turnover industry, like a call center that's hiring, they might view someone who stays a year as a, as a good success, right? Or, or certainly 18 months, right? In technology, you'd have a different view, right? You expect people to stay more than a year if you're so, so it does differ by industry, but I think that that big study that I'm referring to was, was across a lot of different industries. And they, I believe they used a sort of 12 to 18 month sort of window to evaluate. And the, and the basic premise of it was, would you hire this person again, the sort of asking the employer is, do you regret this hire or would you, would you do this again? And that's where they got the 54% figure from.
Yeah, I, I, I love that question. It's a question when we've hired people and we're calling for references, which we don't rely on very much, they're pretty unreliable also. But the question that we do tend to ask, and sometimes we get asked at, of, of former employees of ours is, is if you had a role that was suitable for this person, would you hire them again? I think it's pretty revealing. You know, people grow, people change organizations and the rules change. So it's kind of hard to ask them, would you hire them again for the same role five years later? They're not gonna be the same person. They're not gonna be looking for the same thing. But, you know, I can think back to, to people that we've had that there is absolutely no way that we would ever hire them again. Fortunately the vast majority of people that we've had working for us, absolutely that, that we would, we would hire them and they're good people. They put did good work, and it's just their interest diverged. They were looking for something different where we, we were looking for something different. We'll be back right after this break.
Welcome back to the high volume hiring podcast. You, you, you mentioned about that you don't think that the resume is gonna be going away soon. And the 4,500 customers that you've got around the globe are, are there any that have done away with the resume that they're only using your assessment software, for example? Or are they all basically going dual?
Yeah, I think the vast majority of them are going dual. And it's either a resume or in some cases for certain roles, it's just like an online application, which is similar in certain respects. Like in terms of a data gathering mechanism. It's similar to a resume, right? You're asking for work history, et cetera. So I would say I, I don't know if many that have asked the resume altogether, right? I think that a lot of our customers will use will overweight assessments, right? They'll give it as much weight as resumes. And in fact, a lot of them increasingly are using the assessments at the same point in the hiring process as they collect the resume, right? So very, think of it as very top of funnel, right? They'll use the assessments, which especially a lot of our assessments are designed to be used top of funnel.
So they'll be fairly brief, fairly succinct, like think about a 15 minute assessment, for example. A lot of our customers will use the assessments really at the same point as they collect a resume or immediately after. So, so very early in the hiring process. And the advantage of doing that is that you get an assessment result, whether it's a, you know, an aptitude test or a personality assessment. You get that signal for all applicants or for a very, very large percentage of them. Sometimes employers will use some basic qualifying techniques and then give assessments to everyone who meets the basic criteria. And a lot of them will get them from everyone they get a resume from. And I think your question, that is a sign that they view the resume as a pretty unreliable signal. It's, it's something they want to have. But a a lot of times they're waiting the assessment result much more heavily.
Yeah, we, we did that. We started doing that with, with hiring software developers a couple years ago, where until then we were, we were doing it a traditional way. We were looking at other jobs they'd had, it's the companies, the software they had been using, et cetera. And then we experimented with having a hundred percent of the applicants go through an assessment instead of the, just the finalists. And what was amazing it kind of, I wasn't that surprised because I had talked to some people like you and so I had a, I had a heads up that this is what we were probably gonna find, but some other people on our team were amazed that some of the candidates that on paper on their resume looked to be the best possible candidates of the group assessed the worst, and others that it's like, okay, well she has no work experience that's relevant.
And the assessment was off the charts great. And it just, it, I think it was a really good reminder. People can learn to code on their own and maybe never have been employed in that field. They may have gone through a bootcamp, so they're not gonna have the name of an elite university on, on their resume. And it's still nice to have those resumes, even just as for talking points, you know what, oh, you went to that school. Oh, you lived here. Oh, you worked for that company. Tell me a little bit about the work that you did there. That, that stuff can be helpful even if it's for that. It's
Interesting that you mentioned software developers, by the way, because I, people are listening to this on audio, but I'm no in furiously as Steven is, SA is saying this
Software engineer is one of those positions where, you know, obviously a highly skilled position where there's domain knowledge that's needed. And so experience is often used as a real gatekeeper there. And, and just from a compensation standpoint, someone who's a senior software engineer has five to 10 years experience radical difference in pay, right? From an entry level software engineer. And and yet it's one of the positions where our customers have seen the most success in terms of evaluating people. One of my favorite case studies we have that we did in in the last year is with a customer of ours that's in the, the telemedicine field, which of course really blew up in Covid, right? So a lot of hiring. And they had traditionally asked for their hiring managers had insisted on minimum of three years experience for their software engineers, and they decided to hire more junior entry level engineers as a sort of pilot program.
And it's, it's funny you mentioned boot camps. The two strategies they used where they used our assessments very early in the process, and they focused on boot camps as a sourcing channel not exclusively, but primarily used boot camps. And what they found is I think they hired like 18 people in the, in the first class of these junior software engineers, entry level engineers. And from a dem demographic standpoint, the class looked totally unlike their existing engineering pool. So it was almost half, almost half female, which was very unlike what what they had previously. And it was just demographically very diverse. So a few people over 50 a couple of veterans in the class, they were just sort of getting at groups they had no success at hiring into in, in, in the software engineering group, which is, which had traditionally been in their company sort of twenties and 30 year old men.
And so just from a demographic standpoint, it was really eye-opening for them. And they also had a really high success rate. I think they offered every single person a full-time role after this, this century level sort of apprentice program that they had. So pretty amazing results. And it just, it, for me, I love that story because it's like, it shows how assessments and other tools can really open the aperture of the talent profile you're looking for and widen it, right? And if you go go beyond that three year or that five year insistence on, you know, that much experience, it really can surface talent you, you might not have otherwise noticed, which I, I think is exactly the point you were making.
Yeah. I love that, Josh. It's one of the things that's always frustrated me and, and is about traditional college and university recruiting programs, is that they are, I can't think of any hiring channel that employers have that are more inconsistent with diversity, equity, and inclusion efforts. And because traditionally employers will hire from 10 schools, 20 schools, 30 schools, and maybe one or two or three majors at those schools. So every candidate you hire year after year after year is basically the same. They might have a different skin color they might have a different gender, but they had the same textbooks, they had the same professors, they have the same socioeconomic backgrounds, they all think the same. So, you know, those employers would typically say something like, well, that's where our best employer employees have come from, and so that's why we're going to do that again.
And when you challenge them and say, how do you know that you couldn't have had better employees and maybe the best 20 or 30% came from those schools, but what about the 70% that weren't? Could you hire more of those top 20 or 30%? Are you satisfied with the bottom 10%? And historically the answer back was, well, there's no better way. And maybe 20 years ago that was true. But now there are organizations, you know, like criteria that are using much more scientific methods for identifying people. And how, how likely is it that they're gonna be with you for six months, 18 months, five years? How likely is it that they're gonna be a top performer rather than just what was their degree? What, what's on their diploma? And, and that's really exciting. They listeners that are struggling to sort of align their early career emerging talent, university recruiting, call it what you will programs with your d e I efforts, I got one word for you assessments Josh, I, I, I suspect you don't disagree with that
<Laugh>. Yeah. You're preach to the choir,
You're preaching the choir on that one. And, and I think you go back to that college grad recruiting strategy and it's also relevant, you know, the tight labor market that we're in, you know, I, I was speaking with technology companies recently some of whom were our customers and in a, in a forum. And it's not that the old strategy doesn't work as far as it goes. In other words, if you're hiring engineers, right you will have a very high success rate if you can hire only from Stanford CS programs, right? Like you, you'll probably get a, a very good engineers, right? But that's not a luxury that most companies have, right? There's such intense composition competition for the top CS grads. And so, you know, good luck. You mu you must have a great employer brand if, if you can just fish in that one <laugh>, that one pond, right?
So it's not that that the degree is an unreliable proxy in that case, it's just that the inverse isn't true, right? The fact that someone didn't go to Stanford does not mean that they, they don't have talent. And and in this environment that we're in, in terms of labor market dynamics, and obviously things are shifting a little bit in the technology industry, but more broadly still a very tight labor market with very robust hiring and, and a real long-term labor supply issue. You just need to again, widen your aperture in terms of what you're looking at. And I think that's one reason even the best employer brands and technology several years ago, many of them like Tesla, Facebook, but Meta now dropped, dropped degree requirements, right? It's because they, they have to go more broadly in terms of their search for talent. And if you're looking to surface talent that is not in those few ponds that you know, that everyone's fishing in, you really need tools like assessments to, to help you do that. And so, so I think that's that's what's behind the, the drive to to use assessments more broadly.
Yeah, it's, it's exciting. I I am, I, I I I just think it's a, a really fantastic addition to the talent acquisition tech stack, that it's going to really open up opportunities for employers finding more and better candidates and for the candidates to, to have a better opportunity to, to do the work that they value, that they're good at, that, that matter to them. So Josh, unfortunately we're gonna have to leave off. There are like 18 other questions that I would love to ask you ain't gonna happen. But before we do for the listeners who wanna learn more about criteria Corp or Josh how should they contact you?
Yeah, great. So you can find a lot more information about what we do on our [email protected] You can also find me on LinkedIn Josh millett on, on LinkedIn. So feel free to reach out to me directly there.
Cool. And that's Millett, M I l l E
Right. Awesome. Have a great day. Cheers, Josh. Thanks Steven. Thanks so much.
Thanks for joining us today on the High Volume Hiring podcast. I'm your host, Steven Rothberg of job search Site college recruiter. Each year we help more than 12 million candidates find great new jobs. Our customers are primarily Fortune 1000 companies, government agencies, and other employers who hire at scale and advertise their jobs with us. You can reach me at [email protected] The High Volume hiring podcast is a co-production of Evergreen Podcasts and College Recruiter. Please subscribe for free on your favorite app. Review it five stars are always nice, and recommend it to a couple of people you know who want to learn more about how best to hire at scale. Cheers.