The following function calculates the maximum-sized tile for the given information.
If the fact that it's written in Python makes it hard for you to understand, let me know in a comment and I'll try to do it up in some other language.
import math
from __future__ import division
def max_tile_size(tile_count, rect_size):
    """
    Determine the maximum sized tile possible.
    Keyword arguments:
    tile_count -- Number of tiles to fit
    rect_size -- 2-tuple of rectangle size as (width, height)
    """
    # If the rectangle is taller than it is wide, reverse its dimensions
    if rect_size[0] < rect_size[1]:
        rect_size = rect_size[1], rect_size[0]
    # Rectangle aspect ratio
    rect_ar = rect_size[0] / rect_size[1]
    # tiles_max_height is the square root of tile_count, rounded up
    tiles_max_height = math.ceil(math.sqrt(tile_count))
    best_tile_size = 0
    # i in the range [1, tile_max_height], inclusive
    for i in range(1, tiles_max_height + 1):
        # tiles_used is the arrangement of tiles (width, height)
        tiles_used = math.ceil(tile_count / i), i
        # tiles_ar is the aspect ratio of this arrangement
        tiles_ar = tiles_used[0] / tiles_used[1]
        # Calculate the size of each tile
        # Tile pattern is flatter than rectangle
        if tile_ar > rect_ar:
            tile_size = rect_size[0] / tiles_used[0]
        # Tile pattern is skinnier than rectangle
        else:
            tile_size = rect_size[1] / tiles_used[1]
        # Check if this is the best answer so far
        if tile_size > best_tile_size:
            best_tile_size = tile_size
    return best_tile_size
print max_tile_size(4, (100, 100))
The algorithm can loosely be described as follows
- If the rectangle is higher than it is wide, flip it so that it's wider than it is high.
 
- Calculate s, the square root of the number of tiles, rounded up. (Named 
tiles_max_height in code.) 
- Loop where i goes from 1 to s inclusive:
- Construct a grid of squares that is number of tiles / i squares wide and i squares high. (Round everything up. This "pads" the missing tiles, such as using 2 tiles by 2 tiles when your total number of tiles is 3.)
 
- Make this grid as big as possible. (Calculate this using aspect ratios.) Determine the size of one tile.
 
- Using that size, determine the total area covered by the tiles.
 
- Check if this is the best total area so far; if it is, store the square size
 
 
- Return that square size
 
This is probably one of the faster algorithms listed here, as it computes the best square size in O(sqrt(n)) for n tiles.
Update
On further consideration, this problem has a simpler solution based on the solution above. Say you are given 30 tiles. Your possible tile arrangements are easy to compute:
- 30 x 1 (aspect ratio 30.0000)
 
- 15 x 2 (aspect ratio 7.5000)
 
- 10 x 3 (aspect ratio 3.3333)
 
- 8 x 4 (aspect ratio 2.0000)
 
- 6 x 5 (aspect ratio 1.2000)
 
- 6 x 6 (aspect ratio 1.0000)
 
Say your rectangle is 100 x 60. Your rectangle's aspect ratio is 1.6667. This is between 1.2 and 2. Now, you only need to calculate the tile sizes for the 8 x 4 and the 6 x 5 arrangements.
The first step still technically takes O(sqrt(n)) though, so this updated method is not asymptotically faster than the first attempt.
Some updates from the comments thread
/*
Changes made:
tiles_used -> tiles_used_columns, tiles_used_rows
 (it was originally a 2-tuple in the form (colums, rows))
*/
/* Determine the maximum sized tile possible. */
private function wesleyGetTileSize() : Number {
 var tile_count : Number = slideCount.value;
 var b : Number = heightOfBox.value;
 var a : Number = widthOfBox.value;
 var ratio : Number;    
 // // If the rectangle is taller than it is wide, reverse its dimensions    
 if (a < b) {
  b = widthOfBox.value;
  a = heightOfBox.value;
 } 
 // Rectangle aspect ratio   
 ratio = a / b;    
 // tiles_max_height is the square root of tile_count, rounded up    
 var tiles_max_height : Number = Math.ceil(Math.sqrt(tile_count))    
 var tiles_used_columns : Number;
 var tiles_used_rows : Number;
 var tiles_ar : Number;
 var tile_size : Number;
 var best_tile_size : Number = 0;    
 // i in the range [1, tile_max_height], inclusive   
 for(var i: Number = 1; i <= tiles_max_height + 1; i++) {       
  // tiles_used is the arrangement of tiles (width, height)        
  tiles_used_columns = Math.ceil(tile_count / i);   
  tiles_used_rows = i;
  // tiles_ar is the aspect ratio of this arrangement        
  tiles_ar = tiles_used_columns / tiles_used_rows;        
  // Calculate the size of each tile        
  // Tile pattern is flatter than rectangle       
  if (tiles_ar > ratio){           
   tile_size = a / tiles_used[0]   ;
  }    
  // Tile pattern is skinnier than rectangle        
  else {            
   tile_size = b / tiles_used[1];
  }        
  // Check if this is the best answer so far        
  if (tile_size > best_tile_size){           
   best_tile_size = tile_size;
  }   
 }
 returnedSize.text = String(best_tile_size);
 return best_tile_size;
}