Questions RecOps Folks Should Ask During Their ATS Scoping
Meet the the Author ✍️
Jacob Miller
Jacob was the first sales (and 2nd non-technical hire) at Ashby, where he’s helped companies like Ramp, Docker, Quora, FullStory, and many more, move from a ‘frankenstack’ of tools to Ashby’s All In One Recruiting platform. Previously, Jacob has been an early sales hire at 3 different SaaS startups, where he built and managed high-performing teams and owned all of his own recruiting efforts; an experience which led him to join Ashby!
Jacob currently resides in the East Bay of San Francisco and on a sunny weekend can be found playing the same ultimate frisbee pick up game he’s been a member for the last 15 years - or if Karl the Fog is out in force; you can catch him at natural wine bar most likely.
As Ashby’s first sales hire, I’ve spoken with many TA and TA Ops leaders in the last couple years. I try to act as a strategic consultant, partnering with the buyer to identify with clarity what, how, and why they’re evaluating a solution. My goal is to help people make good decisions; sometimes that means going with Ashby, and sometimes that doesn’t! But either way, I want the buyer to feel they’ve made a fully informed decision, that maps to their specific needs and goals. You can read more about our sales team values here, if you’re generally curious about our approach to sales.
Because I’ve had the unique opportunity to talk to hundreds of buyers - I’ve been able to see certain trends and themes develop, both in terms of things that the best buyers do well, as well as potential blindspots that buyers don’t always know to test for until it’s too late.
This article distills some key questions I wish RecOps professionals knew to ask when evaluating an ATS.
“When do customers typically outgrow you, and where do they go next”?
Most ATS buyers are looking to make a decision that they won’t have to revisit for at least 2-3 years. If a company has aggressive growth plans during that time period, that means it’s important to ‘look around the bend’ and make sure that the product selected can meet today’s needs, but also tomorrow’s.
Some of this can be tested for based on scoping future needs and requirements, and comparing to current product functionality and (potentially) roadmap. But one useful proxy for a vendor’s ability to scale with you = where do their customers usually get off the bus. It’s unusual for a vendor to be uniquely suited to the needs of all of SMB, mid-market, enterprise, strategic, etc. Generally, companies specialize in a particular segment, and customer demographics can be very illustrative of where you can expect their ability to meet customer needs to drop off.
Hopefully your sales rep will be forthcoming about this information; but it may be worthwhile to ask around to your network / community groups.
This kind of context can help prevent decisions I see occasionally where a customer’s stated goal for a project is to implement an ATS that can scale with them, but the product selected is typically outgrown by similar size/ stage companies.
“Can I see a demo focused solely on integrations with our major existing systems (HRIS, Calendar, Email, Zoom, etc.)”
An ATS must integrate, well, with many different systems. Newer buyers tend to treat integrations as a binary “yes/no” conversation - either a vendor has an integration, or they don’t. More experienced buyers (or anyone who’s had to use an ATS with a subpar integration in a key functional area) know better.
Some things to interrogate:
Is the integration out-of-the-box and maintained by the vendor? Or does it require developer resources from the customer?
Is there an additional cost for the integration? One time, or recurring?
Is it one way or two ways?
Is data sync instant? What data does / doesn’t get ported over?
Is it an org level setup or does it require individual setup and authentication by each individual user?
Some specific examples of pain points that we’ve consistently heard around poor integrations.
A “calendar” integration that in practice actually boils down to sending an iCS file to candidates that doesn’t actually connect to interviewer calendars.
An email integration that doesn’t capture emails sent outside the ATS and often misses candidate responses.
An HRIS integration that only pulls over a small handful of offer fields, leading teams to simply abandon the integration in favor of manual record creation.
An e-sign integration that doesn’t support non-offer files types, and/or doesn’t provide visibility to hiring team members on the status of a sent document.
It’s probably worth taking the time to scope out what ideal integration workflows looks like for your needs. Make sure that you avoid assuming that just because a vendor says “oh yeah we integrate with X,” that the integration will actually deliver what you need/want!
🚨 PSA: For those on or considering Workday HRIS - I strongly encourage a session with the ATS you’re evaluating that is solely focused on the topic of their potential integration with Workday HRIS. Oftentimes these integrations are bespoke, custom, very expensive (sometimes on par with the cost of the ATS!), have a long implementation timeline, and often require quite a bit of ongoing maintenance from the customer.
“When it comes to reporting, come to the demo with use cases in mind”
Every ATS you look at will tell you they have great reporting, and can show you some nice looking charts and graphs that would look great in a slide deck. But most fail to meet actual customer needs and it's extremely common for the actual source of reporting truth to live in spreadsheets (highly manual, time consuming, inherent lag time), a BI tool (difficult to set up and maintain, usually requires project managing the data science team for months/quarters at a time - once you’ve gotten access to them), or a 3rd party TA reporting tool (additional cost, system bloat, often difficult to make the numbers line up).
The more astute buyers I work with have scar tissue from working with poor ATS reporting - and that drives the questions they ask during demos. Some examples:
Can I build reports on any field in the ATS?
How robust are filters and groupings on reports? (can I exclude or combine filters? Can any field be used as a filter/grouping, including custom fields?)
“Show me how I could go from looking at this hires by department report, to slicing it by location, by hiring manager, etc. - or filtering it to exclude interns and/or only include hires made from a particular source.”
Basically - how ‘ad-hoc’ friendly is the reporting model. Are you out of luck if you have a question that isn’t covered by a pre-existing template?
What are the underlying assumptions you’re making around complex TA-specific reporting topics like conversion rates, time to hire, etc.?
Example: Some vendors calculate passthrough rates in ways that make them largely unusable for their customers (e.g. by including active candidates in the calculation which artificially lowers conversation rates).
How hardcoded are date ranges on reports?
Super tactical, but almost every time we hear about data trust issues, it boils down to the fact that most ATS reports are ‘hard-coded’ in terms of the date-time stamp - (e.g. there are ‘missing’ hires in a Hires This Quarter report because the ATS is looking at application start date and not ‘when did we have a recruiter screen’ which is what the team actually would like to report on).
What ability do I have to build custom dashboards for completely different audiences (C-Level, hiring managers, Finance/People Ops, Recruiters, etc.)?
“How long will data migration take and what data will be lost”
Most ATS providers will offer to import historical candidate data for their customers. However, there are some nuances to consider here, especially from a timing and resourcing planning perspective.
It’s worth asking specific questions here to understand exactly what would be lost in a potential data migration:
How much data cleanup will be required from the customer, post-migration?
We’ve heard stories about customers needing to manually move candidates from one placeholder job, into the correct jobs, one by one! Which is both massively time-consuming and very detrimental to being able to report on historical hiring speed metrics accurately.
What will be preserved in terms of candidate communication and activity history?
Are you just getting name, email, and perhaps resume? Or will there be a detailed history of feedback forms, email history, notes, etc. to help inform future reviews and/or outreach?
Realistically, what is the timeline from requesting a data import from current provider, to having all of your data ingested into a new ATS system?
We’ve heard of providers taking as long as EIGHT WEEKS to export customer data on request*. And our understanding is a typical file-based migration can take 4-6 weeks to process for other providers. So, if you’re targeting a ‘go-live’ date built around a certain renewal or business milestone, you may want to consider providing up to 2-3 months of buffer solely for the data migration process.
Re the Asterisk: I have to shamelessly plug Ashby’s ability to import data via API for our new customers coming over from Lever or Greenhouse. This doesn’t require the existing vendor to export data, as it’s all handled via the API the customer already has access to - and the migration itself takes 24-72 hours. It is very rich (things that get ported over include notes, email history, files, feedback forms, jobs, stages, sources, etc. etc.)
“What supplemental tools will likely be required should we implement?”
Many vendors make expansive marketing claims to be an “all in one” system… “the only platform you’ll need from source to hire”, etc.
In practice, oftentimes functionality is lacking in key areas like reporting, sourcing, scheduling, surveys, headcount management, etc. There’s a reason why the Recruiting tech ecosystem is flooded with point solutions! These tools are filling gaps in the market where most ATS providers aren’t delivering needed functionality.
If/when conducting customer references; you’ll want to take a holistic approach to questions around their stack of tools, outside the ATS. Are there key workflows that customers are executing outside the platform? What other tools have they needed to purchase and integrate?
It will be important to consider the total cost of ownership as you’re comparing providers. If one solution is cheaper on paper but will require 2-3 point solutions.
Product velocity / agility / responsiveness
The talent space is a very dynamic environment with a lot of pressure placed on TA teams to adapt quickly to changing needs and requirements. In just the past few years, we’re talking about forces such as shift towards remote work + a more global workforce, pay transparency legislation, AI, the Talent Wars, etc. etc.
While digging into a vendor’s roadmap can be helpful, perhaps more meaningful is the vendor’s proven track record of shipping new, relevant features, quickly.
Can they provide a list of all the new features and feature enhancements they’ve released in the last 3, 6, 12 months?
Do their customers feel their feedback is being heard and incorporated into the product?
What kind of investment have they made in terms of their product team - how large is Eng/Product related to the larger company? Are they hiring on the technical side?
G2Crowd reviews can also be helpful here, if a vendor is responsive to customer feedback and feature requests, this will often show up as a common thread in reviews.
To summarize, just because a vendor has all the features you need today; doesn’t mean they will tomorrow! Important to be sure that any vendor you’re working with is ‘skating where the puck is going.’
2024 Addendum
I can’t believe this has to be said, but here we are! If a vendor isn’t using their own ATS for their own internal hiring efforts, that is a HUGE red flag (especially if their company is the same size/stage as their target market). Doesn’t speak well to their ability to continually refine the product with recruiters in mind, or their belief in their own core product.
You’d be surprised how common this is. Pretty easy to check for it - go to their careers page and see what platform is hosting their jobs.
Disclaimer: The views expressed and opinions expressed in this article are those of the author and they do not necessarily reflect the official policy or position of any other agency, organization, employer, or company. Assumptions made in the analysis are not reflective of the position of any entity other than the author. Since we are critically-thinking human beings, these views are always subject to change, revision, and rethinking at any time. Please do not hold them in perpetuity.
