Immediately close the listening socket when a listener is destroyed

By closing the file descriptor early on in the listener destruction process, the listening socket can be reused by other applications. This allows repeated destruction and creation of listeners.