If you want to remove a library manually from the Arduino IDE then you need to find your libraries folder. If your not sure where it is then go to “File->Preferences” and at the top you will see a path for your sketches. For example mine looked like this
C:\Users\Laptop\Documents\XtronicalGraphics
Open this location and within it you will see a folder named “libraries”. Open this and you will find folders named after the libraries you have imported. Simply select and delete as normal. It’s probably a good idea to re-start you Arduino IDE at this point – but not essential generally.