Removing folders has been limited to Administrators only. Regular users and uploaders are not allowed to remove folders, even when they have been given manage rights.
The reason for this is that the folder may have subfolders to which the user/uploader has no access. As you can assign access to folders on all levels, this could otherwise allow a user to remove subfolders of which they are not aware.
Example
For example, consider the following folder structure where we have 2 subfolders in our Customers folder. Our user is the account manager for Customer X and must be able to manage all files for that customer, so he is given manage rights and access to the folder:
[ x ] Customers (access)
.. [ x ] Customer X (access)
.. [ ] Customer Y (no access)
If the user would be able to delete directories, the user could delete the "Customers" directory without having knowledge of the "Customer Y" folder. To keep things easy for our users, we have kept it simple and limited folder removal to Administrators only.