Basic Authentication wasn't designed to manage logging out. You can do it, but not completely automatically.
What you have to do is have the user click a logout link, and send a ‘401 Unauthorized’ in response, using the same realm and at the same URL folder level as the normal 401 you send requesting a login.
They must be directed to input wrong credentials next, eg. a blank username-and-password, and in response you send back a “You have successfully logged out” page. The wrong/blank credentials will then overwrite the previous correct credentials.
In short, the logout script inverts the logic of the login script, only returning the success page if the user isn't passing the right credentials.
The question is whether the somewhat curious “don't enter your password” password box will meet user acceptance. Password managers that try to auto-fill the password can also get in the way here.
Edit to add in response to comment: re-log-in is a slightly different problem (unless you require a two-step logout/login obviously). You have to reject (401) the first attempt to access the relogin link, than accept the second (which presumably has a different username/password). There are a few ways you could do this. One would be to include the current username in the logout link (eg. /relogin?username), and reject when the credentials match the username.