Sounds like you have done everything correct from the SQL Anywhere perspective.
There is no file with the name "dbserv" in the product and there has never been.
I wonder if that is just a stand-in convention that your other vendor is using.
It could be a .cmd or .bat file that they install (or you may need to add or configure)
or it may be shorthand used to indicate to you to supply the real .exe file name
somewhere in the setup. As such you may need to contact that other vendor
to get more details.
On Windows the *server* executible names for SQL Anywhere have always been of the form:
dbsrv17.exe, dbeng16.exe, mlsrv12.exe
for example where the number is version specific. You could try typing in dbsrv17 or dbsrv17.exe
if that is an editible field.
There is a DLL file called dbserv17.dll but I don't believe that is the missing piece here; applications
will never load that directly from there software so don't be confused by that name.
If you get nowhere with the other vendor on this ... you can report back your findings here ... or ...
If this gets too complicated support can be accessed via https://support.sap.com/home.html
Best wishes