I have never been able to wrap my mind around WHY we seem to think that patting a horse or telling it "Good Boy" would be rewarding for a horse. Personally, I think we do it because we find it rewarding. Human language on its own is meaningless sto a horse, and I can't imagine that the horse (or any animal) really wants to be patted - with one of my horses, it would actually be counter-productive since one of my horses is really sensitive to things like that and shies away from them.
Realistically, the only way a pat or a verbal reward could be any sort of reinforcer would be if it was done consistently enough with other things to become associated with those things. For instance, if your horse gets a quick break or a change in activity when they get their pat or "good boy", they could become associated with one another. Essentially, you've done the same thing that clicker training does when it creates a "bridge" between an inherently meaningless reinforcer and gives it meaning through association. However, since you're likely not being consistent and intentionally pairing the real reward with your pat or verbal reward, it probably won't become very strongly associated with any sort of reward that the horse wants to work for.
However, just for kicks and giggles, let's assume that horses find pats and being told "good boy" or "good girl" is very rewarding for a horse.... It would still be a terrible reinforcer, much in the same way simply "treat training" is a terrible reinforcer and for the same reasons. The trouble with treat training is that you cannot give the horse immediate feedback on specific behaviors since it's impossible to give them a treat at that moment. Usually, if it's impossible to to feed a treat, it would probably be impossible to give them a pat. Thus, it's not really connected to the specific behavior you're working on but rather an overall "I did somethingright."
The ultimate test to find out whether your pats or words are real reinforcers would be to stop giving them and keep everything else you're doing exactly the same except. If you stopped patting or saying "good boy", would your horse still work for you at the same level/speed he does now? My bet would be yes - because he's not working for the pat or words. Rather, he's working for the release of pressure, the real reinforcer. Thus, since the pats aren't actually motivating the horse to perform the desired behaviors more often, it, by definition, is not a reinforcer at all.
(Disclaimer - I'm not saying you shouldn't pat/pet/rub your horse or tell them "good boy". In fact, though I don't pat because that just isn't something I do for whatever reason, I do give lots of rubs and verbal "good"-s because I do think it reinforces my relationship with my horse. I don't expect it to assist with my training beyond simply establishing a bond with my horse and being comfortable and happy around each other. In contrast, I expect the reinforcement with the clicker to actually produce results in our training.)