Delete Amazon S3 buckets?

Question

I've been interacting with Amazon S3 through S3Fox and I can't seem to delete my buckets. I select a bucket, hit delete, confirm the delete in a popup, and... nothing happens. Is there another tool that I should use?

Answer 1

I've always ended up using their C# API and little scripts to do this. I'm not sure why S3Fox can't do it, but that functionality appears to be broken within it at the moment. I'm sure that many of the other S3 tools can do it as well, though.

Answer 2

Delete all of the objects in the bucket first. Then you can delete the bucket itself.

Apparently, one cannot delete a bucket with objects in it and S3Fox does not do this for you.

I've had other little issues with S3Fox myself, like this, and now use a Java based tool, jets3t which is more forthcoming about error conditions. There must be others, too.

Answer 3

@Stu - I've tried the delete all the entries thing as well, and it still didn't work. Unless they've recently fixed it, this just seems like a bug.

Answer 4

This may be a bug in S3Fox, because it is generally able to delete items recursively. However, I'm not sure if I've ever tried to delete a whole bucket and its contents at once.

The JetS3t project, as mentioned by Stu, includes a Java GUI applet you can easily run in a browser to manage your S3 buckets: Cockpit. It has both strengths and weaknesses compared to S3Fox, but there's a good chance it will help you deal with your troublesome bucket. Though it will require you to delete the objects first, then the bucket.

Disclaimer: I'm the author of JetS3t and Cockpit

Answer 5

Try RightScale - it's great web app for dealing with AWS.

Answer 6

SpaceBlock also makes it simple to delete s3 buckets - right click bucket, delete, wait for job to complete in transfers view, done.

This is the free and open source windows s3 front-end that I maintain, so shameless plug alert etc.

Answer 7

You must make sure you have correct write permission set for the bucket, and the bucket contains no objects. Some useful tools that can assist your deletion: CrossFTP, view and delete the buckets like the FTP client. jets3t Tool as mentioned above.

Answer 8

With s3cmd: Create a new empty directory s3cmd sync --delete-removed empty_directory s3://yourbucket

Answer 9

I guess the easiest way would be to use S3fm, a free online file manager for Amazon S3. No applications to install, no 3rd party web sites registrations. Runs directly from Amazon S3, secure and convenient.

Just select your bucket and hit delete.

Answer 10

Remeber that S3 Buckets need to be empty before they can be deleted. The good news is that most 3rd party tools automate this process. If you are running into problems with S3Fox, I recommend trying S3FM for GUI or S3Sync for command line. Amazon has a great article describing how to use S3Sync. After setting up your variables, the key command is

./s3cmd.rb deleteall <your bucket name>

Deleting buckets with lots of individual files tends to crash a lot of S3 tools because they try to display a list of all files in the directory. You need to find a way to delete in batches. The best GUI tool I've found for this purpose is Bucket Explorer. It deletes files in a S3 bucket in 1000 file chunks and does not crash when trying to open large buckets like s3Fox and S3FM.

I've also found a few scripts that you can use for this purpose. I haven't tried these scripts yet but they look pretty straightforward.

RUBY

require 'aws/s3'

AWS::S3::Base.establish_connection!(
:access_key_id => 'your access key',
:secret_access_key => 'your secret key'
)

bucket = AWS::S3::Bucket.find('the bucket name')

while(!bucket.empty?)
begin
puts "Deleting objects in bucket"

bucket.objects.each do |object|
object.delete
puts "There are #{bucket.objects.size} objects left in the bucket"
end

puts "Done deleting objects"

rescue SocketError
puts "Had socket error"
end

end

PERL

#!/usr/bin/perl
use Net::Amazon::S3;
my $aws_access_key_id = 'your access key';
my $aws_secret_access_key = 'your secret access key';
my $increment = 50; # 50 at a time
my $bucket_name = 'bucket_name';

my $s3 = Net::Amazon::S3->new({aws_access_key_id => $aws_access_key_id, aws_secret_access_key => $aws_secret_access_key, retry => 1, });
my $bucket = $s3->bucket($bucket_name);

print "Incrementally deleting the contents of $bucket_name\n";

my $deleted = 1;
my $total_deleted = 0;
while ($deleted > 0) {
print "Loading up to $increment keys...\n";
$response = $bucket->list({'max-keys' => $increment, }) or die $s3->err . ": " . $s3->errstr . "\n";
$deleted = scalar(@{ $response->{keys} }) ;
$total_deleted += $deleted;
print "Deleting $deleted keys($total_deleted total)...\n";
foreach my $key ( @{ $response->{keys} } ) {
my $key_name = $key->{key};
$bucket->delete_key($key->{key}) or die $s3->err . ": " . $s3->errstr . "\n";
}
}
print "Deleting bucket...\n";
$bucket->delete_bucket or die $s3->err . ": " . $s3->errstr;
print "Done.\n";

SOURCE: Tarkblog

Hope this helps!

Answer 11

I hacked together a script for doing it from Python, it successfully removed my 9000 objects. See this page:

https://efod.se/blog/archive/2009/08/09/delete-s3-bucket

Answer 12

I'll have to have a look at some of these alternative file managers. I've used (and like) BucketExplorer, which you can get from - surprisingly - http://www.bucketexplorer.com/.

It's a 30 day free trial, then (currently) costing US$49.99 per licence (US$49.95 on the purchase cover page).

Answer 13

One more shameless plug: I got tired of waiting for individual HTTP delete requests when I had to delete 250,000 items, so I wrote a Ruby script that does it multithreaded and completes in a fraction of the time:

http://github.com/sfeley/s3nuke/

This is one that works much faster in Ruby 1.9 because of the way threads are handled.

Answer 14

recent version of s3cmd have --recursive

e.g.,

# s3cmd rb --recursive s3://bucketwithfiles

Answer 15

you also use CloudBerry Explorer freeware to delete the buckets. It works pretty well

Answer 16

Try https://s3explorer.appspot.com/ to manage your S3 account.

Answer 17

Try this freeware : http://www.dragondisk.com

Answer 18

This is what I use. Just simple ruby code.

case bucket.size
  when 0
    puts "Nothing left to delete"
  when 1..1000
    bucket.objects.each do |item|
      item.delete
      puts "Deleting - #{bucket.size} left"        
    end
end

Answer 19

This is a hard problem. My solution is at http://stuff.mit.edu/~jik/software/delete-s3-bucket.pl.txt. It describes all of the things I've determined can go wrong in a comment at the top. Here's the current version of the script (if I change it, I'll put a new version at the URL but probably not here).

#!/usr/bin/perl

# Copyright (c) 2010 Jonathan Kamens.
# Released under the GNU General Public License, Version 3.
# See <http://www.gnu.org/licenses/&gt;.

# $Id: delete-s3-bucket.pl,v 1.3 2010/10/17 03:21:33 jik Exp $

# Deleting an Amazon S3 bucket is hard.
#
# * You can't delete the bucket unless it is empty.
#
# * There is no API for telling Amazon to empty the bucket, so you have to
# delete all of the objects one by one yourself.
#
# * If you've recently added a lot of large objects to the bucket, then they
# may not all be visible yet on all S3 servers. This means that even after the
# server you're talking to thinks all the objects are all deleted and lets you
# delete the bucket, additional objects can continue to propagate around the S3
# server network. If you then recreate the bucket with the same name, those
# additional objects will magically appear in it!
# 
# It is not clear to me whether the bucket delete will eventually propagate to
# all of the S3 servers and cause all the objects in the bucket to go away, but
# I suspect it won't. I also suspect that you may end up continuing to be
# charged for these phantom objects even though the bucket they're in is no
# longer even visible in your S3 account.
#
# * If there's a CR, LF, or CRLF in an object name, then it's sent just that
# way in the XML that gets sent from the S3 server to the client when the
# client asks for a list of objects in the bucket. Unfortunately, the XML
# parser on the client will probably convert it to the local line ending
# character, and if it's different from the character that's actually in the
# object name, you then won't be able to delete it. Ugh! This is a bug in the
# S3 protocol; it should be enclosing the object names in CDATA tags or
# something to protect them from being munged by the XML parser.
#
# Note that this bug even affects the AWS Web Console provided by Amazon!
#
# * If you've got a whole lot of objects and you serialize the delete process,
# it'll take a long, long time to delete them all.

use threads;
use strict;
use warnings;

# Keys can have newlines in them, which screws up the communication
# between the parent and child processes, so use URL encoding to deal
# with that. 
use CGI qw(escape unescape); # Easiest place to get this functionality.
use File::Basename;
use Getopt::Long;
use Net::Amazon::S3;

my $whoami = basename $0;
my $usage = "Usage: $whoami [--help] --access-key-id=id --secret-access-key=key
 --bucket=name [--processes=#] [--wait=#] [--nodelete]

    Specify --processes to indicate how many deletes to perform in
    parallel. You're limited by RAM (to hold the parallel threads) and
    bandwidth for the S3 delete requests.

    Specify --wait to indicate seconds to require the bucket to be verified
    empty. This is necessary if you create a huge number of objects and then
    try to delete the bucket before they've all propagated to all the S3
    servers (I've seen a huge backlog of newly created objects take *hours* to
    propagate everywhere). See the comment at the top of the script for more
    information about this issue.

    Specify --nodelete to empty the bucket without actually deleting it.\n";

my($aws_access_key_id, $aws_secret_access_key, $bucket_name, $wait);
my $procs = 1;
my $delete = 1;

die if (! GetOptions(
       "help" => sub { print $usage; exit; },
       "access-key-id=s" => \$aws_access_key_id,
       "secret-access-key=s" => \$aws_secret_access_key,
       "bucket=s" => \$bucket_name,
       "processess=i" => \$procs,
       "wait=i" => \$wait,
       "delete!" => \$delete,
 ));
die if (! ($aws_access_key_id && $aws_secret_access_key && $bucket_name));

my $increment = 0;

print "Incrementally deleting the contents of $bucket_name\n";

$| = 1;

my(@procs, $current);
for (1..$procs) {
    my($read_from_parent, $write_to_child);
    my($read_from_child, $write_to_parent);
    pipe($read_from_parent, $write_to_child) or die;
    pipe($read_from_child, $write_to_parent) or die;
    threads->create(sub {
 close($read_from_child);
 close($write_to_child);
 my $old_select = select $write_to_parent;
 $| = 1;
 select $old_select;
 &child($read_from_parent, $write_to_parent);
      }) or die;
    close($read_from_parent);
    close($write_to_parent);
    my $old_select = select $write_to_child;
    $| = 1;
    select $old_select;
    push(@procs, [$read_from_child, $write_to_child]);
}

my $s3 = Net::Amazon::S3->new({aws_access_key_id => $aws_access_key_id,
          aws_secret_access_key => $aws_secret_access_key,
          retry => 1,
         });
my $bucket = $s3->bucket($bucket_name);

my $deleted = 1;
my $total_deleted = 0;
my $last_start = time;
my($start, $waited);
while ($deleted > 0) {
    $start = time;
    print "\nLoading ", ($increment ? "up to $increment" :
    "as many as possible")," keys...\n";
    my $response = $bucket->list({$increment ? ('max-keys' => $increment) : ()})
 or die $s3->err . ": " . $s3->errstr . "\n";
    $deleted = scalar(@{ $response->{keys} }) ;
    if (! $deleted) {
 if ($wait and ! $waited) {
     my $delta = $wait - ($start - $last_start);
     if ($delta > 0) {
  print "Waiting $delta second(s) to confirm bucket is empty\n";
  sleep($delta);
  $waited = 1;
  $deleted = 1;
  next;
     }
     else {
  last;
     }
 }
 else {
     last;
 }
    }
    else {
 $waited = undef;
    }
    $total_deleted += $deleted;
    print "\nDeleting $deleted keys($total_deleted total)...\n";
    $current = 0;
    foreach my $key ( @{ $response->{keys} } ) {
 my $key_name = $key->{key};
 while (! &send(escape($key_name) . "\n")) {
     print "Thread $current died\n";
     die "No threads left\n" if (@procs == 1);
     if ($current == @procs-1) {
  pop @procs;
  $current = 0;
     }
     else {
  $procs[$current] = pop @procs;
     }
 }
 $current = ($current + 1) % @procs;
 threads->yield();
    }
    print "Sending sync message\n";
    for ($current = 0; $current < @procs; $current++) {
 if (! &send("\n")) {
     print "Thread $current died sending sync\n";
     if ($current = @procs-1) {
  pop @procs;
  last;
     }
     $procs[$current] = pop @procs;
     $current--;
 }
 threads->yield();
    }
    print "Reading sync response\n";
    for ($current = 0; $current < @procs; $current++) {
 if (! &receive()) {
     print "Thread $current died reading sync\n";
     if ($current = @procs-1) {
  pop @procs;
  last;
     }
     $procs[$current] = pop @procs;
     $current--;
 }
 threads->yield();
    }    
}
continue {
    $last_start = $start;
}

if ($delete) {
    print "Deleting bucket...\n";
    $bucket->delete_bucket or die $s3->err . ": " . $s3->errstr;
    print "Done.\n";
}

sub send {
    my($str) = @_;
    my $fh = $procs[$current]->[1];
    print($fh $str);
}

sub receive {
    my $fh = $procs[$current]->[0];
    scalar <$fh>;
}

sub child {
    my($read, $write) = @_;
    threads->detach();
    my $s3 = Net::Amazon::S3->new({aws_access_key_id => $aws_access_key_id,
       aws_secret_access_key => $aws_secret_access_key,
       retry => 1,
      });
    my $bucket = $s3->bucket($bucket_name);
    while (my $key = <$read>) {
 if ($key eq "\n") {
     print($write "\n") or die;
     next;
 }
 chomp $key;
 $key = unescape($key);
 if ($key =~ /[\r\n]/) {
     my(@parts) = split(/\r\n|\r|\n/, $key, -1);
     my(@guesses) = shift @parts;
     foreach my $part (@parts) {
  @guesses = (map(($_ . "\r\n" . $part,
     $_ . "\r"   . $part,
     $_ . "\n"   . $part), @guesses));
     }
     foreach my $guess (@guesses) {
  if ($bucket->get_key($guess)) {
      $key = $guess;
      last;
  }
     }
 }
 $bucket->delete_key($key) or
     die $s3->err . ": " . $s3->errstr . "\n";
 print ".";
 threads->yield();
    }
    return;
}