on the sly - definition and synonyms

phrase informal
  1. done secretly, especially because you know you should not be doing it

    I’m supposed to be on a diet, but I still have the odd bar of chocolate on the sly.

See also main entry: sly