The main thing to remember is that hex, octal, binary, and decimal are just different ways of representing same integers. So, yes, you need to send the decimal values of the integers to the serial object and everything should work.
That said, the documentation you posted does take a bit of figuring out which bits need to be set how, and from thence converting them to integer values. Also, a possible trap may be making sure you send bytes to the [serial] object in the order your device expects to receive them.
Also make sure you've got the bit rate, parity, etc. matching.
Serial can be a bit of a PITA to set up, but once it works it's fairly straightforward.