Not initialising granulepos leads to it having arbitrary values in the encoded stream including possibly negative values which are not valid and confuse opusdec. Explicitly initialise opus_encoder::granulepos to avoid that problem.
.. and move the plugins to src/encoder/plugins/.