CheckBox changes value twice

Question

I have an Android ListView whose items have a checkbox.

The checkbox is checked by default. Once unchecked it should be removed from the list.

The problem is that onCheckedChanged is being fired twice: when I tap the checkbox to uncheck it (with isChecked false) and after I remove the item (with isChecked true).

This is the relevant code of my ArrayAdapter:

public View getView(final int position, View convertView, ViewGroup parent) {
    ViewHolder holder;
    if (convertView == null) {
        convertView = mInflater.inflate(R.layout.item, parent, false);
        holder = new ViewHolder();
        holder.check = (CheckBox) convertView.findViewById(R.id.check);
        convertView.setTag(holder);
    } else {
        holder = (ViewHolder) convertView.getTag();
    }
    final Object item = this.getItem(position);
    holder.check.setOnCheckedChangeListener(new CheckBox.OnCheckedChangeListener() {
        @Override
        public void onCheckedChanged(CompoundButton buttonView, boolean isChecked) {
            if (!isChecked) {
                remove(item); // This somehow calls onCheckedChanged again
            }
        }
    });
    return convertView;
}

What am I doing wrong?