Keep database synced between computers in Trellis/Bedrock/Sage setup

Ok, I dug a little deeper in the VVV approach and tried to replicate their database syncing method with the following steps:

  1. Added the following directory structure in my main project directory:

database/
├─ backups/
├─ data/
├─ import-sql.sh
├─ init-custom.sql.sample (not used)
└─ init.sql (not used)

  1. Changed the sync folder in the wordpress_sites loop in my Vagrantfile into:

    config.vm.synced_folder “…/database/”, "/srv/database"
    if File.exists?(File.join(’…/database/data/mysql_upgrade_info’)) then
    config.vm.synced_folder “database/data/”, “/var/lib/mysql”, :mount_options => [ “dmode=777”, “fmode=777” ]
    end

  2. Added the following directory structure and files in my ansible directory:

config/
├─ homebin/
├── db_backup
├── vagrant_destroy
├── vagrant_halt
└── vagrant_suspend

and added the following to the Vagrantfile:

 config.vm.synced_folder "config/", "/srv/config"
  1. Added a provision directory in the ansible directory with an edited version of the VVV’s provision.sh file:

    #!/bin/bash

    provision.sh

    This file is specified in Vagrantfile and is loaded by Vagrant as the primary

    provisioning script whenever the commands vagrant up, vagrant provision,

    or vagrant reload are used. It provides all of the default packages and

    configurations included with Varying Vagrant Vagrants.

    If MySQL is installed, go through the various imports and service tasks.

    exists_mysql="$(service mysql status)"
    if [[ “mysql: unrecognized service” != “${exists_mysql}” ]]; then

     # IMPORT SQL
     # Setup MySQL by importing an init file that creates necessary
     # users and databases that our vagrant setup relies on.
     # mysql -u root -proot < /srv/database/init.sql
     # echo "Initial MySQL prep..."
    
     # Process each mysqldump SQL file in database/backups to import
     # an initial data set for MySQL.
     /srv/database/import-sql.sh
    

    else
    echo -e "\nMySQL is not installed. No databases imported."
    fi

and added the following to the Vagrantfile:

if File.exists?(File.join(ANSIBLE_PATH,'provision','provision.sh')) then
  config.vm.provision :shell, :path => File.join( "provision", "provision.sh" )
end
  1. Installed the vagrant-triggers plugin and added the following at the bottom of my Vagrantfile:

    if Vagrant.has_plugin? 'vagrant-triggers’
    config.trigger.before :halt, :stdout => true do
    run "vagrant ssh -c ‘vagrant_halt’"
    end
    config.trigger.before :suspend, :stdout => true do
    run "vagrant ssh -c ‘vagrant_suspend’"
    end
    config.trigger.before :destroy, :stdout => true do
    run "vagrant ssh -c ‘vagrant_destroy’"
    end
    else
    puts 'vagrant-triggers missing, please install the plugin:'
    puts 'vagrant plugin install vagrant-triggers’
    end

Running vagrant up doesn’t give any errors besides stdin: is not a tty

==> default: Running provisioner: shell...
    default: Running: /var/folders/x8/l1j6fgb91s37px9tplwmfdmdn2c64s/T/vagrant-shell20150824-7848-13ekkph.sh
==> default: stdin: is not a tty
==> default: 
==> default: Start MySQL Database Import
==> default: No custom databases to import

No databases are imported yet, but that makes sense because one has to be exported first on vagrant halt

That’s where I’m stuck right now, the vagrant_destroy, vagrant_halt & vagrant_suspend commands are called from within the vagrant ssh in the vagrant-triggers but how does Vagrant ‘know’ that these commands are stored inside the config/homebin directory?

When running vagrant halt I get the following error right now:

==> default: Running triggers before halt...
==> default: Executing command "vagrant ssh -c vagrant_halt"...
==> default: bash: vagrant_halt: command not found
==> default: Command execution finished.
The command "vagrant ssh -c 'vagrant_halt'" returned a failed exit code. The
error output is shown below:

bash: vagrant_halt: command not found

How do I ‘load’ these commands into my Vagrantfile?

Thanks again and sorry for the long post!

3 Likes

I got a bit closer!

When I run vagrant ssh and run the following command (where example_dev, username & password are defined in your group_vars/development file):

mysqldump -uusername -ppassword example_dev > /srv/database/backups/example_dev.sql

My database is exported correctly and synced in my local directory database/backups!

But I need to insert this command into the run command of the vagrant trigger instead of using the external command (config/homebin/db_backup) like VVV does within a wordpress_sites loop like this:

 if Vagrant.has_plugin? 'vagrant-triggers'
  config.trigger.before :halt, :stdout => true do
    wordpress_sites.each do |(name, site)|
      db_name  = site['env']['db_name']
      db_user  = site['env']['db_user']
      db_pass  = site['env']['db_password']
      info "Backing up #{db_name}"
      run "vagrant ssh -c echo #{db_name} && mysqldump -u#{db_user} -p#{db_pass} #{db_name} > /srv/database/backups/#{db_name}.sql && echo 'Database #{db_name} backed up';"
    end
  end
else
  puts 'vagrant-triggers missing, please install the plugin:'
  puts 'vagrant plugin install vagrant-triggers'
end

But I keep on getting this error:

The command "vagrant ssh mysql -u username -p password
      mysqldump -uusername -ppassword  example_dev > /srv/database/backups/example_dev.sql" returned a failed exit code. The
error output is shown below:

An invalid option was specified. The help for this command
is available below.

Usage: vagrant ssh [options] [name] [-- extra ssh args]

Options:

    -c, --command COMMAND            Execute an SSH command directly
    -p, --plain                      Plain mode, leaves authentication up to user
    -h, --help                       Print this help

I already tried it without the -c flag but that doesn’t make a difference.
What exactly is the invalid option?

Thanks!

2 Likes

I’m excited that you’re figuring out cool stuff here. I cheer you on, while also saying that I probably won’t engage on it much more given that it is outside of regular Trellis and not a personal need of my own.

I couldn’t help wondering though, have you already tried vagrant ssh -c 'command in quotes'
When you omit quotes, maybe vagrant is thinking that the parts of the COMMAND are supposed to be options, and they look like invalid options.

Yes, thank you!
I think I got it working now! :grinning:

# Vagrant Triggers
if Vagrant.has_plugin? 'vagrant-triggers'
  
  # Importing Database
  wordpress_sites.each do |(name, site)|
    db_name  = site['env']['db_name']
    db_user  = site['env']['db_user']
    db_pass  = site['env']['db_password']
    {
      [:up, :resume, :reload] => "vagrant ssh -c 'cd /srv/database/backups/
      sql_count=`ls -1 #{db_name}.sql 2>/dev/null | wc -l`
      if [ $sql_count != 0 ]
      then
        echo Database file exists, so start importing
        mysql -u #{db_user} -p#{db_pass} #{db_name} < /srv/database/backups/#{db_name}.sql
      else
        echo Database file does not exist  
      fi'",
    }.each do |command, trigger|
      config.trigger.after command, :stdout => true do
        info "Importing database #{db_name}"
        run  "#{trigger}" 
      end
    end
  end
  
  # Exporting Database
  wordpress_sites.each do |(name, site)|
    db_name  = site['env']['db_name']
    db_user  = site['env']['db_user']
    db_pass  = site['env']['db_password']
    {
      [:halt, :suspend, :destroy] => "vagrant ssh -c 'mysqldump -u #{db_user} -p#{db_pass} #{db_name} > /srv/database/backups/#{db_name}.sql'",
    }.each do |command, trigger|
      config.trigger.before command, :stdout => true do
        info "Backing up #{db_name}"
        run  "#{trigger}"
      end
    end
  end  
else
  puts 'vagrant-triggers missing, please install the plugin:'
  puts 'vagrant plugin install vagrant-triggers'
end

On vagrant up, resume and reload my database file gets imported correctly if it exists and on vagrant halt, suspend and destroy it gets exported correctly!

If anyone sees any potential bugs or problems, please let me know!
Thanks for all your help @fullyint!

5 Likes

Oh by the way, in order to get this up and running;
On all the computers you want to sync, your MacOs user id should be the same (and probably your user name too). Here’s how you can change this: https://support.apple.com/en-us/HT201548

Make sure to create a second Administrator user first and log in with that new account first before changing the User id and/or name. After changing the home folder name and id and logging back in, my account was all messed up. I had to change the file permissions on my home folder and reinstall Dropbox, Vagrant and Virtualbox in order to get things working again!

But it was worth it, because it’s working now on 2 computers!

2 Likes

Dear Toine ,

I am a newbie in Vagrant and I have made a script which install LAMP but what I missed is the sync of the database can you please share the entire script on how you sync the database?

It would be really much appreciated.

Marco

Hi Marco,

Here’s my fork of the Vagrantfile:
https://github.com/Twansparant/trellis/blob/master/Vagrantfile

Besides installing vagrant-triggers, that’s the only file you need to edit.

Good luck!

2 Likes

Thank you so much … ;o))

Marco

I found out if you exclude the following folder in your Trellis clone from your Dropbox sync:

trellis // Your clone of the Trellis repository
└──.vagrant/
     └── machines/
          └── default/
               └── virtualbox/

There’s no need to change your MacOs user id since this id is stored in the creator_uid file.
This saves you a lot of work…

I now also have excluded my vendor, node_modules & bower_components directories which speeds things up a bit in the Dropbox syncing.

3 Likes

hey @Twansparant, this looks fantastic, I’m attempting to see if I can get this rolling as well. It looks like the Trellis Vagrantfile is pretty drastically different. It looks like most of your implementation is just related to Vagrant Triggers right? Ideally I could use the upstream version of that Vagrantfile and install your triggers modification to that. Swapping the entire Vagrantfile leads to some odd changes in the VM.

Yeah and you need an extra synched folder for the database:
config.vm.synced_folder "../database/", "/srv/database"[quote=“qbunt, post:30, topic:4478”]
Swapping the entire Vagrantfile leads to some odd changes in the VM.
[/quote]
Hmm that shouldn’t happen, I’m using this exact same Vagrantfile without any problems?
What kind of changes?

@Twansparant the modification actually could have been caused by another dev unbeknownst to me. However, from what I can tell, it looks like this SSH command:

vagrant ssh -c 'cd /srv/database/backups/
        sql_count=`ls -1 #{db_name}.sql 2>/dev/null | wc -l`
        if [ $sql_count != 0 ]
        then
          echo Database file exists, so start importing
          mysql -u #{db_user} -p#{db_pass} #{db_name} < /srv/database/backups/#{db_name}.sql
        else
          echo Database file does not exist  
        fi'"

…is either not getting executed by the vagrant machine or the host machine isn’t sending it through correctly. I’m using zsh as my shell, so that could be adding unnecessary troubles. The reason I asked about the exact modifications is that the upstream Vagrantfile and the one in your Gist have a fair amount of differences seemingly unrelated to migrating databases.

Going to try just running this directly in bash and hopefully remove zsh from the equation, hopefully that helps.

Ok, don’t know anything about zsh so not sure if that could be causing it?

My Vagrantfile is based upon Trellis 0.9.1, could be that some non-database-related changes are made, I haven’t updated to 0.9.3 yet.

Let me know how that goes!

Ok I just tried my custom vagrant-triggers and can confirm it doesn’t work anymore with Trellis 0.9.5 and up.
The problem is, the trigger commands need to read the database credentials in the wordpress_sites loop, but since the db_password variable is moved to the (encrypted) vault.yml, the database dump or import won’t succeed because there is no password set in the command. So I’m trying something like this now:

config_file = File.join(ANSIBLE_PATH, 'group_vars', 'development', 'wordpress_sites.yml')
vault_file = File.join(ANSIBLE_PATH, 'group_vars', 'development', 'vault.yml')

if File.exists?(config_file)
  wordpress_sites = YAML.load_file(config_file)['wordpress_sites']
  vaults = YAML.load_file(vault_file)['vaults'] --vault-password-file .vault_pass
  #vaults = ansible-vault YAML.load_file(vault_file)
  fail_with_message "No sites found in #{config_file}." if wordpress_sites.to_h.empty?
else
  fail_with_message "#{config_file} was not found. Please set `ANSIBLE_PATH` in your Vagrantfile."
end

# Vagrant Triggers
#
# If the vagrant-triggers plugin is installed, we can run various scripts on Vagrant
# state changes like `vagrant up`, `vagrant halt`, `vagrant suspend`, and `vagrant destroy`
#
# These scripts are run on the host machine, so we use `vagrant ssh` to tunnel back
# into the VM and execute things.
#
if Vagrant.has_plugin? 'vagrant-triggers'

  vaults.each_pair do |vault|
    #
    # Get database password
    #
    db_pass  = vault['env']['db_password']
  end

  wordpress_sites.each_pair do |site|
    #
    # Get database credentials
    #
    db_name  = site['env']['db_name']
    db_user  = site['env']['db_user']
    #
    # Importing database
    #
    config.trigger.after [:up, :resume, :reload], :force => true do
      info "Importing databases"
      run_remote  "cd /srv/database/backups/ mysql -u #{db_user} -p#{db_pass} #{db_name} < /srv/database/backups/#{db_name}.sql"
    end
    #
    # Exporting database
    #
    config.trigger.before [:halt, :suspend, :destroy], :force => true do
      info "Dumping databases"
      run_remote "mysqldump -u #{db_user} -p#{db_pass} #{db_name} > /srv/database/backups/#{db_name}.sql"
    end
  end
else
  puts 'vagrant-triggers missing, please install the plugin:'
  puts 'vagrant plugin install vagrant-triggers'
end

But that’s obviously not working yet…

How exactly can I load and set the db_password variables from the encrypted group_vars/development/vault.yml file?
http://docs.ansible.com/ansible/playbooks_vault.html#viewing-encrypted-files

Or is this not gonna work like this?
Any help is welcome, thanks!

1 Like

Ok, for now I’m not going to encrypt my groups_vars/development/vault.yml file to be able to read the db_password variables. However, since I’m not a Ruby expert;

At the top of my Vagrantfile I set the vault_wordpress_sites variable like this:

vault_file = File.join(ANSIBLE_PATH, 'group_vars', 'development', 'vault.yml')

if File.exists?(vault_file)
  vault_wordpress_sites = YAML.load_file(vault_file)['vault_wordpress_sites']
  fail_with_message "No sites found in #{vault_file}." if vault_wordpress_sites.to_h.empty?
else
  fail_with_message "#{vault_file} was not found. Please set `ANSIBLE_PATH` in your Vagrantfile."
end

Somehow I need to read the db_password variable from this vault_wordpress_sites array inside the existing wordpress_sites loop, something like this (but with correct syntax):

wordpress_sites.each_pair do |name, site|
      #
      # Get database credentials
      #
      db_name  = site['env']['db_name']
      db_user  = site['env']['db_user']
      #db_pass  = vault_wordpress_sites.#{index}.env.db_password
end

But I have no clue how to approach this… Anyone familiar with Ruby loops?
Thanks!

Got a bit further, loop is working now:

vault_passes = Array.new
vault_wordpress_sites.each_pair do |name, vault|
  db_pass  = vault['env']['db_password']
  vault_passes.push db_pass
end

wordpress_sites.each_with_index do |(name, site), index|
  #
  # Get database credentials
  #
  db_name  = site['env']['db_name']
  db_user  = site['env']['db_user']
  db_pass  = vault_passes[index]
  #
  # Importing database
  #
  config.trigger.after [:up, :resume, :reload], :force => true do
    info "Importing databases"
    run_remote "cd /srv/database/backups/ mysql -u #{db_user} -p#{db_pass} #{db_name} < /srv/database/backups/#{db_name}.sql"
  end
  #
  # Exporting database
  #
  config.trigger.before [:halt, :suspend, :destroy], :force => true do
    info "Dumping databases"
    run_remote "mysqldump -u #{db_user} -p#{db_pass} #{db_name} > /srv/database/backups/#{db_name}.sql"
  end
end

You can simplify this a lot:

config_file = File.join(ANSIBLE_PATH, 'group_vars', 'development', 'wordpress_sites.yml')
vault_file = File.join(ANSIBLE_PATH, 'group_vars', 'development', 'vault.yml')

vault_sites = YAML.load_file(vault_file)['vault_wordpress_sites']

wordpress_sites.each do |(name, site)|
  site['env'].merge!(vault_sites[name]['env'])

  # triggers in here
end
1 Like

Thanks, gonna try that next time!

Yup, that works great, thanks!
My updated triggers look like this now:

vault_file = File.join(ANSIBLE_PATH, 'group_vars', 'development', 'vault.yml')
vault_sites = YAML.load_file(vault_file)['vault_wordpress_sites']

# Vagrant Triggers
#
# If the vagrant-triggers plugin is installed, we can run various scripts on Vagrant
# state changes like `vagrant up`, `vagrant halt`, `vagrant suspend`, and `vagrant destroy`
#
# These scripts are run on the host machine, so we use `vagrant ssh` to tunnel back
# into the VM and execute things.
#
if Vagrant.has_plugin? 'vagrant-triggers'    
  wordpress_sites.each do |(name, site)|
    #
    # Get database credentials
    #
    site['env'].merge!(vault_sites[name]['env'])
    db_name  = site['env']['db_name']
    db_user  = site['env']['db_user']
    db_pass  = site['env']['db_password']
    #
    # Importing database
    #
    config.trigger.after [:up, :resume, :reload], :force => true do
      info "Importing databases"
      run_remote "mysql -u #{db_user} -p#{db_pass} #{db_name} < /srv/database/backups/#{db_name}.sql"
    end
    #
    # Exporting database
    #
    config.trigger.before [:halt, :suspend, :destroy], :force => true do
      info "Dumping databases"
      run_remote "mysqldump -u #{db_user} -p#{db_pass} #{db_name} > /srv/database/backups/#{db_name}.sql"
    end
  end
else
  puts 'vagrant-triggers missing, please install the plugin:'
  puts 'vagrant plugin install vagrant-triggers'
end
4 Likes

I have setup Trellis with an existing WP site. I’d like to use this solution to sync the database. But, I’m getting the following error message:

/trellis/Vagrantfile:145:in block (2 levels) in <top (required)>': undefined method 'merge!' for nil:NilClass (NoMethodError)

Just started to search around for solution. Thought I would post here first to see if anyone else has experienced this problem.