A first class job would use a 2 part linear polyurethane, such as awlgrip, rolled on, with a sand type additive sprinkled on, or sprayed on, and then locked in with more paint, perhaps with flattening agent.
There are some good single part paints too. Epifanes and interlux both make some. They already have non skid mixed into the paint. They can be rolled on easily and quite uniformally. They won't have the durability of the 2 part paints. This would be an inferior finish compared to awlgrip. But is all in what you want. I've got single part on my little boat, and it's great. But I know it won't get serious wear and tear.
Is it for the cockpit sole or for the top of gunnel across the transom? The gunnel portion across the back of the transom is what I call the stern deck, just want to know which part of the boat your referring to.