Great question! I just double-checked and you are completely right, there is currently no build-in way to completely remove a downloaded dictionary.
What you can do is the following:
1. Go to "Download" -> "Recent"
2. Note the path to the directory (or file) of the dictionary
3. Then long-press on the dictionary you don't need and tap on "Remove from list"
Now you can use any third-party file manager to navigate to the noted path and delete it completely, for example OI File Manager
(free and no ads).
Hope that helps,