An individual recording or a single folder (and all its contents) can be deleted from the Context Menu in Just Press Record on iOS 13 or later.
Folders and files can be deleted in bulk using the iOS Files app if you are using using the iCloud Drive storage option. If you have Just Press Record set to use the Local Storage, recordings will not show in the Files app so this will not be possible.