The book is about algorithms, not the details of programming, and the language of choice is just to make the examples concrete. As the author says, "The code examples in this book are written in Python... but I provide explanations of all the algorithms so that programmers of other languages can follow." (p. xv)
Python is a great language and easy to learn, but I suspect the difficulties in applying ideas from the book will not be in the translating of the code to another language or set of libraries, but in understanding the ideas and modifying the code to suite your needs. I think there are two main reasons to stay with a language you're familiar with: 1) when your code doesn't work, if you're writing in an unfamiliar language, you won't know where to start looking for errors, e.g. if you're like most people you'll even start wondering if it's due to a bug in Python, which it won't be, but you'll wonder and it will distract. 2) There are just natural limits to how much you can remember in a certain length of time; and learning a language at the same time will give you twice as much to remember.
It depends though how well you know C#, and what you lose by leaving it.