In this paper, we propose novel deterministic and probabilistic public key cryptographic schemes based on an effective and compact subgroup trace representation cryptosystem to handle with the problem of secure and efficient communication between the server and resource-constrained device. The proposed schemes use the hardness of the Trace-discrete logarithmic like problem. We also show that the deterministic version of the proposed scheme is a one-way trapdoor, and the probabilistic version of the proposed scheme is semantically secure. Moreover, we discuss the efficiency of the proposed schemes by comparing with effective and compact subgroup schemes. Copyright (C) 2015 John Wiley & Sons, Ltd.