Roots Discourse

Hoping someone might be able to help with integrating the Ansible Playbook with Amazon CodeDeploy


#1

Hey guys,

This might be beyond the scope of this forum, but I’m hoping that maybe someone has enough know how with the Bedrock-Ansible setup, and some code samples I’m about to show you, and can tell me what I’m supposed to do here.

My problem is that I’m deploying to multiple Amazon EC2 instances in an Auto Scaling Group. Deploying to each one is a pain, as there might be as many as 2-6 (or more) instances running at once, with instances being terminated, or being spun up all the time. I’ve written many scripts and cron jobs to basically sync an s3 bucket with my /var/www/example.com folder so that each instance has the same one. That works ok, but any time I need to update something it can be a pain. Not to mention its far from fool proof.

One way to solve it would be to use Capistrano and an ec2-plugin that deploys to autoscaling groups automatically. The only problem is that Bedrock-Ansible is set up to use Ansible deploys, so it would kind of defeat the purpose of having to set up Capistrano.

Now Amazon has this great CodeDeploy tool that not only will solve the problem, but allows for so much more.

I’ve read on their site that integration with existing Ansible Playbooks should be simple enough, and they even gave me some sample configurations. The only problem is, I don’t have the slightest idea what to do with them. I only got to marginally understanding the ins and outs of Ansible, enough to get by and to the point I’m currently at, so right now my brain hurts looking through the sample configs below and trying to figure out where they would go.

Here is the page that is linked from the AWS CodeDeploy page for Ansible integration- https://github.com/awslabs/aws-codedeploy-samples/tree/master/conf-mgmt/ansible/

According to their site “If you already have a set of Ansible playbooks, but just need somewhere to run them, the template for Ansible and AWS CodeDeploy demonstrates how a couple of simple deployment hooks will ensure Ansible is available on the local deployment instance and run the given playbooks. Alternatively, if you already have a process for building and maintaining your inventory, there’s also an Ansible module that you can use to install and run the AWS CodeDeploy Agent.” (http://docs.aws.amazon.com/codedeploy/latest/userguide/integrations.html)

So… does anyone have any idea how I might want to set this up? Any help at all is always appreciated!

Thanks,
P.J.


#2

I haven’t really looked into CodeDeploy and I agree that their Ansible sample is a little confusing.

But it seems to me like your problems would be solved by just using an AWS inventory script. By default we just manually define hosts/IPs in the hosts folder. So yeah that would get really annoying to constantly manually update those with the hostnames of your autoscaled servers.

But Ansible can work with dynamic inventory scripts and they provide an EC2 one. http://docs.ansible.com/intro_dynamic_inventory.html is what you need it seems like.

Basically instead of running ansible-playbook -i hosts/production deploy.yml you run ansible-playbook -i ec2.py deploy.yml


#3

Thanks for getting back to me.

I had no idea that script even existed. I just glanced over that page, but so far, this might be close to what I’m looking for. I"M HOPING, that what I think I’m reading is correct, and that I can set this script to deploy to any and all instances I’ve given a certain tag to. If thats the case, this does exactly what the Capistrano-ec2 plugin does. Now comes the part the hard part where I actually try and set it up.

Thanks again. You really do go above and beyond the call of duty for your work.


#4

What I also can’t seem to find anywhere is the correct way to set up deploying to s3, like you would with ec2 in the hosts/production file, if I were to go that direction to simplify things.


#5

Wait like deploying static assets to s3?

If no: doesn’t make any sense. Look at what s3 does again.

If yes: look at how many assets you have. What actual problem are you having?

If your problem is you need to host assets on a CDN for perf: have cloudfront proxy your regular server as the origin.

If your problem is you have over a gig of assets and need a place to store them: probably look into some plugins/s3 file system.

If none of the two above: you don’t have a problem.


#6

I’ve been using s3 for over a year as a shared folder for my /var/www/mysite.com folder for all of my ec2 instances.

I have cron jobs set up to pull down changes from the folder on s3 to the folder on all of my ec2 instance every 2 minutes so that every instance has the same /var/www/mysite.com folder. This was fantastic, but a lot has changed since then, and like I said earlier, its not fool proof. It also doesn’t fit my workflow well, especially now that I’m using Ansible to provision my servers.

I’d like a way to deploy that folder to s3 just like if I were to deploy it to ec2. The deploy.yml basically does just that, deploys the site folder, so I want to set up my hosts/production file to send to s3 instead of an ec2 instance or server. I just can’t find the correct way to code it, if its possible at all. I know I can control s3 with Ansible, but I just can’t figure out the correct way to code it in the host file.

I have plenty of plugins to backup/share assets on s3. Thats not the problem. Its making sure every ec2 instance that is spun up has the same folders, so that when I make changes, I don’t have to individually change them all. Hense why I’m trying to find an automation that fits my workflow best, without having to add much more in the way of resources.

This is just one possible way for me to achieve what I’m trying to do, and I’m using it as a backup plan in case the idea swalkinshaw gave me… I’m crossing my fingers, but so far it seems to be working… so far… I just got it to echo a list of all my instances with a certain tag, so I’m assuming I’m on the right track to being able to deploy to them!

Thanks guys.


#7

I’ve gotten to the server.yml script to work correctly to provison all of my instances with a certain tag, now the problem is that because its not using an environment when I go to use deploy.yml I get

TASK: [deploy | Initialize] *************************************************** 
fatal: [52.7.246.149] => One or more undefined variables: 'site' is undefined

deploy.sh obviously uses variables to call the site, but I don’t know how to define it if I’m using the ec2 script and a custom host in group_vars called tag_Site_Mysitename (which includes everything that was in Production).

Any ideas?


#8

So instead of having group_vars/production you have group_vars/tag_Site_Mysitename?

deploy.sh only uses the environment name to call the necessary hosts file. In your case since you’re using the inventory script you should just have to modify deploy.sh but it’s probably just easier to call the command manually:

ansible-playbook -i ec2.py deploy.yml --extra-vars="site=<sitename>"

If the provisioning with server.yml worked then that means the wordpress_sites variables are being picked up properly. No reason why deploy.yml wouldn’t work either.


#9

Now that you’re deploying with Git via Ansible you don’t need to do this S3 syncing/uploading of your entire site folder. It’s just redundant and unnecessary. Just stick with one deployment method.

If you need to share your uploads between servers (which I assume you do), then I’d just use a WordPress plugin to do specifically that.



#10

I mean I’m sure this has worked well but you are seeing the limitations. You should not be syncing assets between servers. There is no reason to do this. Just have cloudfront proxy the s3 server as the origin and then use that URL to prefix all your assets that are hosted.


#11

@peterjohnjoseph don’t listen to this kid

edit: nvm listen to him


#12

I don’t need the S3 method anymore if this works. It was one or the other. I needed a backup, and going back to my old method of deploying to an s3 bucket, and then syncing each instance with it via cron jobs was just my old standby that I learned from taking an AWS course back in the day.

And austin, I appreciate the help, but “that kid” built all of this…


#13

@swalkinshaw is my boy. Lol quote from Austin Pray:

probably look into some plugins

Great minds etc. etc.


#14

Actually Scott is just the face of bedrock-ansible because he was the prettiest


#15

If those statements arnt a boost to your ego, I dont know what is.

Ok, I almost have this thing working… My browser tabs looks like the equivalent to a college student with all of his books laid out on the floor cramming for finals, I have no idea how or why I did things that I tried hours ago, or how I even got to the point I’m at, but if it works I hope it helps someone at some point.

Anyone happen to know if the

    import boto
ImportError: No module named boto

error is common? This is just another part thats kept me up all night. I have boto installed for sure, and Ive seen other people with similar issues, but nothing has worked… I sure as hell hope If this works it helps someone else in the future. Thanks guys. I may of been up for hours, but I’d probably be even further behind if not for the help.


#16

When do you get this error? I’m assuming the ec2.py inventory script has/does work since you said you can provision to your servers. Obviously that means it can find the boto module at that point.


#17

I’ve been getting this error all night when I try to run the ec2.yml script as anything but root. Boto is installed. From what Im reading its a common problem. I just made some progress figuring it out, and it looks like it has the do with its path, so hopefully I can get back to trying to deploy.


#18

Probably depends how you installed boto and Ansible but there’s a bunch of options I’ve seen to fix including two here: https://github.com/ansible/ansible/issues/5734

Good luck


#19

Thats the exact page I was using! Turns out it had nothing to do with any of that… My path for Python was pointing to my MAMP application folder. Somehow that PATH variable took precedence. Just had to remove it.

Ok, now back to deploying… I’m really hoping I’m almost over the last hurdle here. I’d like some sleep, but I’m on a deadline. If it works I’ll post my configurations in here in case anyone ever searched for something similar like I did. Thanks again for all your help, Like I said, its above and beyond the call of duty.


#20

No problem! I was hoping you could get this working and show your work since I haven’t heard of anyone else doing this yet.