Backup / Mirror Github repositories

Question

Hi all,

I'd like to periodically create a backup of my github repositories. Is there a quick way to pull all of them without knowing what the entire list is?

Walter

Answer 1

You can get the entire list via GitHub's API:

curl http://github.com/api/v2/yaml/repos/show/walterjwhite

For example, this simple DOS/Unix shell one-liner:

ruby -ryaml -ropen-uri -e "puts YAML.load(open('http://github.com/api/v2/yaml/repos/show/walterjwhite'))['repositories'].map {|r| %Q[* **#{r[:name]}** (#{r[:description]}) is at <#{r[:url]}/>] }"

prints (assuming you have Ruby installed):

• logback (The reliable, generic, fast and flexible logging framework for Java.) is at http://github.com/walterjwhite/logback/
• Test-Automation (provides UI Test Automation framework) is at http://github.com/walterjwhite/Test-Automation/
• model.base (Base Model Project) is at http://github.com/walterjwhite/model.base/
• project.configuration (Project Configuration (Jalopy, Checkstyle, etc.)) is at http://github.com/walterjwhite/project.configuration/
• model.http (Http Models / Enties) is at http://github.com/walterjwhite/model.http/
• model.address (Address models and entities) is at http://github.com/walterjwhite/model.address/
• model.geolocation.base (Geolocation models and entities) is at http://github.com/walterjwhite/model.geolocation.base/
• model.user (User models and entities) is at http://github.com/walterjwhite/model.user/
• model.annotations.current (Used to inject the current date / user into an entity) is at http://github.com/walterjwhite/model.annotations.current/
• model.annotations.deidentify (Used to deidentify PHI fields) is at http://github.com/walterjwhite/model.annotations.deidentify/
• model.annotations.verify (Used to verify fields (email address, address, phone number, etc.)) is at http://github.com/walterjwhite/model.annotations.verify/
• model.time (Time entities) is at http://github.com/walterjwhite/model.time/
• model.http-security (Http Security entities) is at http://github.com/walterjwhite/model.http-security/
• model.restriction.base (Base Restriction Models) is at http://github.com/walterjwhite/model.restriction.base/
• model.restriction.time (Time Restrictions) is at http://github.com/walterjwhite/model.restriction.time/
• model.restriction.user (User/Group/Role Restrictions) is at http://github.com/walterjwhite/model.restriction.user/
• model.web.base (Base Web Model) is at http://github.com/walterjwhite/model.web.base/
• model.web.resource (Web Resource Model Project) is at http://github.com/walterjwhite/model.web.resource/
• model.web.file (File model) is at http://github.com/walterjwhite/model.web.file/
• model.web.content (Web Content Models) is at http://github.com/walterjwhite/model.web.content/
• model.deployment (Deployment models / entities) is at http://github.com/walterjwhite/model.deployment/
• model.geolocation.tracking (Geolocation Tracking) is at http://github.com/walterjwhite/model.geolocation.tracking/
• model.geolocation.device-location (Device Location models) is at http://github.com/walterjwhite/model.geolocation.device-location/
• model.defect (Defect models / entities) is at http://github.com/walterjwhite/model.defect/
• model.test.error (Test Errors) is at http://github.com/walterjwhite/model.test.error/
• model.web.automation.history (Test History) is at http://github.com/walterjwhite/model.web.automation.history/
• model.html (Test Html) is at http://github.com/walterjwhite/model.html/
• model.test.user (Test Http) is at http://github.com/walterjwhite/model.test.user/
• model.web.automation.test (Test Test entities) is at http://github.com/walterjwhite/model.web.automation.test/
• model.web.automation.validation (Test Validation entities) is at http://github.com/walterjwhite/model.web.automation.validation/
• service.query.base (Provides framework for automatically injecting queries from an interface definition + annotation) is at http://github.com/walterjwhite/service.query.base/
• service.query.defect (Provides query service for querying against defects / fixes) is at http://github.com/walterjwhite/service.query.defect/
• service.query.deployment (Deployment query service) is at http://github.com/walterjwhite/service.query.deployment/
• model.restriction.geolocation (Geolocation restrictions) is at http://github.com/walterjwhite/model.restriction.geolocation/
• service.query.address (Address query servicd) is at http://github.com/walterjwhite/service.query.address/
• service.query.geolocation.device-location (Device Location Query Service) is at http://github.com/walterjwhite/service.query.geolocation.device-location/
• service.query.geolocation.base (Geolocation Query Service) is at http://github.com/walterjwhite/service.query.geolocation.base/
• service.query.http (Http Query Service) is at http://github.com/walterjwhite/service.query.http/
• service.query.http-security (Http Security query service) is at http://github.com/walterjwhite/service.query.http-security/
• service.query.user (User Query Service) is at http://github.com/walterjwhite/service.query.user/
• service.query.web.file (Web File Query Service) is at http://github.com/walterjwhite/service.query.web.file/
• service.query.web.content (Web Content Query Service) is at http://github.com/walterjwhite/service.query.web.content/
• observer.startup.deployment (Startup Event Observer (deployment)) is at http://github.com/walterjwhite/observer.startup.deployment/
• observer.startup.defect (Startup Event Observer (defect)) is at http://github.com/walterjwhite/observer.startup.defect/
• listener.servlet.request (Request Servlet Listener) is at http://github.com/walterjwhite/listener.servlet.request/
• util.network (Network utilities) is at http://github.com/walterjwhite/util.network/
• listener.servlet.geolocation (Geolocation Servlet Listener) is at http://github.com/walterjwhite/listener.servlet.geolocation/
• listener.persistence.audit (Audit Persistence Listeners (Hibernate)) is at http://github.com/walterjwhite/listener.persistence.audit/
• listener.persistence.current (Current (User / Date) Persistence Listeners) is at http://github.com/walterjwhite/listener.persistence.current/
• listener.persistence.restriction (Restriction Persistence Listener (secures entities on load)) is at http://github.com/walterjwhite/listener.persistence.restriction/
• service.geolocation (Geolocation update service) is at http://github.com/walterjwhite/service.geolocation/
• service.notification (Notification Service) is at http://github.com/walterjwhite/service.notification/
• service.traffic-analysis (Traffic Analysis Service) is at http://github.com/walterjwhite/service.traffic-analysis/
• servlet.file (File Servlet) is at http://github.com/walterjwhite/servlet.file/
• util.web.file (File Utilities) is at http://github.com/walterjwhite/util.web.file/
• servlet.filter.base (Base Servlet Filters) is at http://github.com/walterjwhite/servlet.filter.base/
• servlet.filter.web.content (Web Content servlet filter) is at http://github.com/walterjwhite/servlet.filter.web.content/
• servlet.filter.security (Security servlet filters) is at http://github.com/walterjwhite/servlet.filter.security/
• io.file (File functions (deprecated)) is at http://github.com/walterjwhite/io.file/
• deprecated.listener.postInitialization.libraryLogger (Library Logger (deprecated)) is at http://github.com/walterjwhite/deprecated.listener.postInitialization.libraryLogger/
• deprecated.listener.startup.databaseImporter (Database Importer (allows multiple import.sql scripts to be run on server start)) is at http://github.com/walterjwhite/deprecated.listener.startup.databaseImporter/
• deprecated.listener.startup.scheduler (Scheduler (enables jobs to be schedule simply with annotations)) is at http://github.com/walterjwhite/deprecated.listener.startup.scheduler/
• docbook (Docbook format (provides docbook configuration for standard formatting)) is at http://github.com/walterjwhite/docbook/
• deprecated.model.client (Client models) is at http://github.com/walterjwhite/deprecated.model.client/
• deprecated.model.virtualhost (Virtualhost models) is at http://github.com/walterjwhite/deprecated.model.virtualhost/
• util.web.resource (Web Resource utils (deprecated, use security listeners instead for transparent security of entities)) is at http://github.com/walterjwhite/util.web.resource/
• servlet.filter.notification (Notification servlet filter) is at http://github.com/walterjwhite/servlet.filter.notification/
• spring-core (base project for spring projects (deprecated)) is at http://github.com/walterjwhite/spring-core/
• view.content (JSF components for content projects (provides the view layer)) is at http://github.com/walterjwhite/view.content/
• documentation.server-setup (Documentation for setting up a server (trying to make a repeatable process and ensure fewer defects and potential vulnerabilities)) is at http://github.com/walterjwhite/documentation.server-setup/
• server.http (Http Server using JBoss Netty) is at http://github.com/walterjwhite/server.http/
• listener.persistence.deidentify (Deidentification persistence listener) is at http://github.com/walterjwhite/listener.persistence.deidentify/
• util.http (Http Utilities) is at http://github.com/walterjwhite/util.http/
• deprecated.service.translate (Translation Service) is at http://github.com/walterjwhite/deprecated.service.translate/
• html5.example.danforys.mesmerizer (HTML5 grid example) is at http://github.com/walterjwhite/html5.example.danforys.mesmerizer/
• html5.framework (HTML5 framework (caching, client renderering, etc.)) is at http://github.com/walterjwhite/html5.framework/
• documentation.os.open-solaris (Open Solaris Documentation) is at http://github.com/walterjwhite/documentation.os.open-solaris/
• documentation.development (Development Documentation) is at http://github.com/walterjwhite/documentation.development/
• interceptor.exception (Generic exception interceptor for CDI) is at http://github.com/walterjwhite/interceptor.exception/
• observer.http-security (Http Security Observer (authentication events)) is at http://github.com/walterjwhite/observer.http-security/
• model.preference.base (Base Preference model project) is at http://github.com/walterjwhite/model.preference.base/
• model.preference.notification (Notification Preference Models) is at http://github.com/walterjwhite/model.preference.notification/
• deprecated.web.base (Deprecated project (abstract web components)) is at http://github.com/walterjwhite/deprecated.web.base/
• model.annotations.ui (Annotations for automatically rendering Web Action Entities) is at http://github.com/walterjwhite/model.annotations.ui/
• model.web.action.geolocation (Geolocation web action models) is at http://github.com/walterjwhite/model.web.action.geolocation/
• listener.servlet.web.traffic (Web Traffic Servlet Listener (captures web traffic)) is at http://github.com/walterjwhite/listener.servlet.web.traffic/
• model.web.traffic (Web Traffic models (for capturing HTML5 traffic, can be used for HTML4 as well)) is at http://github.com/walterjwhite/model.web.traffic/
• iterator.sql (SQL Iterator (for iterating through SQL scripts)) is at http://github.com/walterjwhite/iterator.sql/
• model.notification (Notification Models/Entities) is at http://github.com/walterjwhite/model.notification/
• model.fmea (FMEA models (for capturing potential failure mode and effects analysis)) is at http://github.com/walterjwhite/model.fmea/
• service.web.automation.operator (Operator for web automation project (simulates an application user)) is at http://github.com/walterjwhite/service.web.automation.operator/
• deprecated.iterator.web-automation (Web Automation iterators (page navigation)) is at http://github.com/walterjwhite/deprecated.iterator.web-automation/
• deprecated.OS.gentoo.install (Gentoo Install scripts (deprecated)) is at http://github.com/walterjwhite/deprecated.OS.gentoo.install/
• web.action.base (Base Web Action module) is at http://github.com/walterjwhite/web.action.base/
• web.action.contact.email (Provides the web action for contacting the web site via email) is at http://github.com/walterjwhite/web.action.contact.email/
• web.action.geolocation (Provides geolocation web action (type in an ip address, get back a geolocation)) is at http://github.com/walterjwhite/web.action.geolocation/
• web.action.search (Search web action (ability to search through all entities provided the user has permission)) is at http://github.com/walterjwhite/web.action.search/
• web.action.user (User web actions (for managing users)) is at http://github.com/walterjwhite/web.action.user/
• web.service.geolocation (Geolocation web service) is at http://github.com/walterjwhite/web.service.geolocation/
• service.query.web.resource (Web Resource query service) is at http://github.com/walterjwhite/service.query.web.resource/
• model.web.action.contact.email (Email backing the Contact action) is at http://github.com/walterjwhite/model.web.action.contact.email/
• interceptor.web (intercepts web action/service methods for auditing purposes) is at http://github.com/walterjwhite/interceptor.web/
• prettyfaces.extension.sitemap (Prettyfaces sitemap extension (automatically generates a sitemap and publishes it to google, yahoo, msn, etc.)) is at http://github.com/walterjwhite/prettyfaces.extension.sitemap/
• prettyfaces-tests (Some PrettyFaces test webapps) is at http://github.com/walterjwhite/prettyfaces-tests/
• scheduler.quartz (Quartz Job Scheduler manager (initializes a quartz job scheduler ...)) is at http://github.com/walterjwhite/scheduler.quartz/
• observer.rFactor (rFactor event observers (for managing an rFactor server)) is at http://github.com/walterjwhite/observer.rFactor/
• observer.event-planner (Event Planner observers) is at http://github.com/walterjwhite/observer.event-planner/
• web.action.news (News web action components) is at http://github.com/walterjwhite/web.action.news/
• web.action.rFactor (rFactor web actions (managing rFactor servers, configurations, races, etc.)) is at http://github.com/walterjwhite/web.action.rFactor/
• web.action.contact.chat (Chat web action components) is at http://github.com/walterjwhite/web.action.contact.chat/
• service.automation.web.search.job (Job Search Web Automation Service) is at http://github.com/walterjwhite/service.automation.web.search.job/
• deprecated.service.analytics (analytics service (deprecated, use an actual reporting package instead)) is at http://github.com/walterjwhite/deprecated.service.analytics/
• service.backup (Backup Service (backs up a database, log files, etc.)) is at http://github.com/walterjwhite/service.backup/
• model.web.action.bookmark (bookmark web action models) is at http://github.com/walterjwhite/model.web.action.bookmark/
• model.web.action.search (search models) is at http://github.com/walterjwhite/model.web.action.search/
• model.web.action.decision (Decisicion models (for capturing decisions and outcomes, useful for design decisions, and of course endless possibilities)) is at http://github.com/walterjwhite/model.web.action.decision/
• model.weather (weather models) is at http://github.com/walterjwhite/model.weather/
• model.recipe (recipe models) is at http://github.com/walterjwhite/model.recipe/
• model.web.action.search.job (Job Search web action models) is at http://github.com/walterjwhite/model.web.action.search.job/
• model.web.action.task (Task web action models) is at http://github.com/walterjwhite/model.web.action.task/
• model.travel (Travel Models) is at http://github.com/walterjwhite/model.travel/
• model.events (Events model) is at http://github.com/walterjwhite/model.events/
• model.lodging (Lodging model) is at http://github.com/walterjwhite/model.lodging/
• model.business.billing (Billing Business Models (for capturing billing information)) is at http://github.com/walterjwhite/model.business.billing/
• model.business.business (Business Models (for capturing billing information)) is at http://github.com/walterjwhite/model.business.business/
• model.business.service (Service Business Models (for capturing services provided)) is at http://github.com/walterjwhite/model.business.service/
• model.business.restaurant (Restaurant Business Models (for capturing billing information)) is at http://github.com/walterjwhite/model.business.restaurant/
• model.news (News models) is at http://github.com/walterjwhite/model.news/
• model.project (Project models (for tracking projects, work, tasks, etc.)) is at http://github.com/walterjwhite/model.project/
• model.meal (Meal models) is at http://github.com/walterjwhite/model.meal/
• model.gaming.rFactor (rFactor game models) is at http://github.com/walterjwhite/model.gaming.rFactor/
• model.question-and-answer (Question and Answer models) is at http://github.com/walterjwhite/model.question-and-answer/
• model.feedback (Feedback models) is at http://github.com/walterjwhite/model.feedback/
• linuxtag2010_javaEE6 (Example code of Java EE 6 tutorial at Linuxtag 2010.) is at http://github.com/walterjwhite/linuxtag2010_javaEE6/
• deprecated.navigation.web.automation (Navigation Web automation module) is at http://github.com/walterjwhite/deprecated.navigation.web.automation/
• model.career (Career models / entities) is at http://github.com/walterjwhite/model.career/
• model.clothing (Clothing models / entities) is at http://github.com/walterjwhite/model.clothing/
• deprecated.model.crawler (deprecated, these projects will be split up (reference purposes only)) is at http://github.com/walterjwhite/deprecated.model.crawler/
• resume (My Resume) is at http://github.com/walterjwhite/resume/
• business-cards (Business Cards) is at http://github.com/walterjwhite/business-cards/

Answer 2

The answer I was waiting for.

I decided to give Ruby a try and it is okay. I like how it is compact, but it isn't pretty looking :(.

This works:

#!/usr/bin/env ruby
require "yaml"
require "open-uri"

time = Time.new
backupDirectory = "/storage/backups/github.com/#{time.year}.#{time.month}.#{time.day}"
username = "walterjwhite"

#repositories =
# .map{|r| %Q[#{r[:name]}] }

#FileUtils.mkdir_p #{backupDirectory}

YAML.load(open("http://github.com/api/v2/yaml/repos/show/#{username}"))['repositories'].map{|repository|

    puts "found repository: #{repository[:name]} ... downloading ..."
    #exec
    system "git clone [email protected]:#{username}/#{repository[:name]}.git #{backupDirectory}/#{repository[:name]}"
}

Walter