My first thought is why not buy a thumbdrive with hardware encryption? ($$$, I know)
Portable apps has a version of VeraCrypt, but it says it does require Admin rights to run.
Portable apps also has a version of GPG, which wouldn't need apps.
Finally, what about installing Portable Apps version of 7z on the thumbdrive, using that to compress and encrypt a folder, and then just using the Portable app version of 7z to open the archive?
That way everything is all on the thumb drive, encrypted, and doesn't require elevated privileges.