There's no way to "upgrade" the software like in the old days, where you could buy an upgrade licence. Today you'll have to purchase new SQL Anywhere 16 licences according to your needs.
Technically you will have to rebuild any existing SQLA 5 database to a new SQLA 16 database using Sybase Central (Tools» SQL Anywhere 16» Unload Database) or the DBUNLOAD utility. In most cases we've had no trouble reloading older databases to the new structure. But there might be some issues with SQL syntax that is not supported anymore, so your mileage may vary.