Ming,
that's a tricky question. First of all you've to chose the right engine. If you're using InnoDB you can set the Buffer-pool and there will be all the data cached. What data? Everithing. Data, indeed, indexes, metainformation about the table, etc. Remember, for InnoDB, the buffer-pool.
For MyISAM, the cache is splitted into diferent buffers. For example, you can set the key_buffer_size , and there MyISAM will cache the keys information.
For your particular example, in the first query it will scan all the table. If you have a buffer_pool big enough, it will be in memory. It's the same for the second one.
Other thing. You should allways have an index for that kind of querys.
Interesting links about buffer_pool:
http://www.mysqlperformanceblog.com/2007/11/03/choosing-innodb_buffer_pool_size/
http://dev.mysql.com/doc/refman/5.0/en/innodb-tuning.html