If everything in scrum is all about functional things that a user can see is there really any place for refactoring code unrelated to any new functional requirements?
I think you are probably talking about large scale refactoring rather than the continuous refactoring you would do whilst in the whole red-green-refactor cycle.
My approach would be something like this, if reafactoring an old feature makes it easier to add a new feature then go ahead and do it. But in some ways you are right, if there is no pressure on a particular unit to change (i.e. it is completely finished and will never change again and will never impact on other modules) then there is no practical need to refactor. However I rarely find a module that is quite so finalised.
I think there is a fair case to make for technical debt refactoring where the effort/cost impact of maintaining the code is as high as, or higher even, than the cost of refactoring it to improve quality or work better / properly - specifically to lend it a higher degree of maintainability.
eg: if the software is so problematic you are losing customers, or money, you'd act fast to fix it.. Some might argue this is a business requirement of it's own, but it's often not placed front and centre on small to mid sized development projects, which instead focus on the technicalities of creating apps rather than the impact of the quality of the app on the bottom line.
I don't think that this has as much to do with Scrum as it does with project management philosophy.
Regardless of whether a project uses Scrum or not, many project managers do not like developers spending time on "unnecessary" things like code refactoring or restructuring that doesn't directly advance one of the outstanding functional requirements. It's not "work that yields results" like normal development, it's "work that prevents a delay of results later". Given the typically short time-lines used for Sprints, the benefit is often hard to see and nearly impossible to quantify.
Keeping code maintainable needs to be an item on your burn-down list (if you use a Scrum). It is just as important as new development. While it may not seem like something that is "visible to the user", ignoring it increases your technical debt. Down the road when the technical debt piles up enough that your code's lack of maintainability slows down development, the delays in new feature development will be visible to customers.
It's all a matter of management/philosophy. Instead of looking at refactoring and maintainability enhancements as "extra" work that doesn't impact customers, it should be viewed as a time investment to prevent customer-visible delays (and potentially bugs as well) down the road. Developers can sometimes see these benefits more clearly than managers can; if your manager doesn't understand the disadvantages of neglecting maintainability, you might want to grab several other developers and have a chat with your manager.
If everything in Scrum is all about functional things that a user can see (...)
Any project and methodology should be about generating business value, you rarely do things just for the fun in a business environment. Having that said, I see quality in Scrum (and other Agile methods) as a way to not kill your velocity on the long run and, ultimately to achieve hyper productivity. I thus believe that a typical "Definition of Done" should include something like "no increase of technical debt" (put your quality standards in there). If you think a new feature will impact existing code that should be refactored, include this cost in the estimate (or create a refactoring item in your Product Backlog) and explain things to your Product Owner. Because at the end, it's up to the Product Owner to prioritize items and to decide if quality can be sacrificed temporarily (if your business die because you don't release a feature, what is the point of refactoring existing code?). But he must be aware that this can't be a long term strategy or he will kill the team velocity.
If you can justify it as part of the process of completing other tasks by identifying issues/risks with current sets of code, and it is a better end result, go for it. But don't get overzealous and screw the timelines/budget.