[{"content":"","date":"November 10, 2026","externalUrl":null,"permalink":"/categories/blog/","section":"Categories","summary":"","title":"Blog","type":"categories"},{"content":"","date":"November 10, 2026","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"November 10, 2026","externalUrl":null,"permalink":"/tags/engineering/","section":"Tags","summary":"","title":"Engineering","type":"tags"},{"content":"","date":"November 10, 2026","externalUrl":null,"permalink":"/tags/engineering-manager/","section":"Tags","summary":"","title":"Engineering-Manager","type":"tags"},{"content":"","date":"November 10, 2026","externalUrl":null,"permalink":"/","section":"Ken Cochrane","summary":"","title":"Ken Cochrane","type":"page"},{"content":"","date":"November 10, 2026","externalUrl":null,"permalink":"/tags/kiss/","section":"Tags","summary":"","title":"Kiss","type":"tags"},{"content":"","date":"November 10, 2026","externalUrl":null,"permalink":"/post/","section":"Posts","summary":"","title":"Posts","type":"post"},{"content":" Three engines, one lesson # There\u0026rsquo;s a photo SpaceX released of its three generations of Raptor rocket engines, lined up side by side (it\u0026rsquo;s the one at the top of this post). I keep coming back to it.\nRaptor 1, on the left, looks like a plate of chrome spaghetti. Pipes loop over pipes, sensors and wiring hang off every surface, and it\u0026rsquo;s hard to tell where one component ends and the next begins. Raptor 2, in the middle, is cleaner but still busy. Raptor 3, on the right, looks almost unfinished by comparison: a smooth, dark body with a handful of clean lines. If you didn\u0026rsquo;t know better, you might think someone forgot to bolt the rest of it on.\nThat photo is the best picture I\u0026rsquo;ve ever seen of something I\u0026rsquo;ve spent my whole career trying to do.\nSimpler, and better on every number # The clean-looking engine isn\u0026rsquo;t a cosmetic redesign. It\u0026rsquo;s the best engine of the three by every measure SpaceX published:\nEngine Sea-level thrust Engine mass Engine + vehicle-side hardware Raptor 1 185 tf 2,080 kg 3,630 kg Raptor 2 230 tf 1,630 kg 2,875 kg Raptor 3 280 tf 1,525 kg 1,720 kg That last column is the one I find most striking. Raptor 3 makes about 50% more thrust than Raptor 1, and the total mass it asks the rocket to carry dropped by more than half. Because so much plumbing moved inside the engine, it no longer needs the heavy heat shield its predecessors carried. Fewer parts to protect means fewer parts, period. (NASASpaceflight, citing SpaceX\u0026rsquo;s published specs)\nWhy we\u0026rsquo;re drawn to the complicated one # Here\u0026rsquo;s a thought experiment. Show those three engines to people who don\u0026rsquo;t follow rockets and ask which one is the best. I\u0026rsquo;d bet most of them pick Raptor 1.\nIt looks like more engine. More pipes, more parts, more going on. We tend to read visible complexity as a sign of capability and effort. Raptor 3 looks like it does less, so it must be the cheap model.\nThe same instinct shows up everywhere. A dashboard with forty charts looks more serious than one with three. An architecture diagram covered in boxes and arrows looks more sophisticated than one you can explain in a sentence. A 60-page design doc feels like more work than a two-page one, even when the two-page one was harder to write.\nThat\u0026rsquo;s a trap, and it\u0026rsquo;s one of the reasons so few people put in the work to make things simple. Simplicity doesn\u0026rsquo;t advertise the effort behind it. If you do it well, nobody notices you did anything at all.\nThe complexity didn\u0026rsquo;t disappear. Someone absorbed it. # Raptor 3 is not a simpler machine than Raptor 1 in any physical sense. It\u0026rsquo;s still a full-flow staged combustion engine running at enormous pressures. What changed is where the complexity lives. Much of the external plumbing was moved inside the engine body or eliminated. Cooling channels are built into the walls. Parts that used to be bolted together are now one piece.\nThat took years of engineering and two full generations of learning. The engineers had to understand Raptor 1 deeply enough to know which parts were essential and which were just scaffolding for problems they hadn\u0026rsquo;t solved yet.\nThis is the part people miss about simple systems. Simple is not the starting point. It\u0026rsquo;s what you get after you\u0026rsquo;ve wrestled with the complicated version long enough to really understand it. The first version of almost anything is a Raptor 1, and that\u0026rsquo;s fine. The mistake is stopping there and calling it done, or worse, being proud of how complicated it looks.\nAntoine de Saint-Exupéry put it well: perfection is achieved not when there is nothing more to add, but when there is nothing left to take away.\nWhat this looks like in software # I saw this up close in the early days of Docker. Linux containers already existed. The kernel features were there: namespaces, cgroups, layered filesystems. But using them meant stitching together a pile of low-level tools and config that only a handful of people understood. It was a Raptor 1.\nDocker didn\u0026rsquo;t invent a more powerful container. It took all of that machinery and hid it behind docker run. Overnight, millions of developers could use something that had been the domain of kernel experts. The complexity was all still there. Someone had just done the hard work of absorbing it so nobody else had to.\nI see the same pattern now with GitOps. Plenty of teams have deployment pipelines that look like Raptor 1: a tangle of scripts, manual steps, and tribal knowledge that one or two people know how to run. GitOps replaces that with a simple idea: the desired state of your system lives in Git, and a tool like Argo CD makes reality match it. Want to deploy? Merge a pull request. Want to roll back? Revert a commit.\nNone of that is simple to build. Reconciling thousands of resources across hundreds of clusters is hard. But the person deploying their app shouldn\u0026rsquo;t have to know any of that. That\u0026rsquo;s the point.\nThe best software I\u0026rsquo;ve worked on doesn\u0026rsquo;t look impressive from the outside. It looks obvious. That\u0026rsquo;s how you know someone did the work.\nEvery junior engineer builds a Raptor 1 # I see this pattern most often with junior engineers, and I say that with affection, because I was one too.\nJunior engineers are excited to build things and try new technology. So a simple app ends up with Kafka, Elasticsearch, Redis, PostgreSQL, MongoDB, and RabbitMQ, when PostgreSQL alone, or PostgreSQL plus a cache, would have been fine. But where\u0026rsquo;s the fun in that?\nThey don\u0026rsquo;t know what they don\u0026rsquo;t know yet. Every component you add comes with a bill, and it keeps arriving long after launch:\nOperations. Every one of those services has to be up, monitored, backed up, and scaled. If any one fails, the whole system can fail with it. Six components can mean six single points of failure. Cost. Each one needs its own infrastructure, and often its own managed service or license. Brain share. Someone has to understand how every piece works and how they all interact. That knowledge is expensive to build and easy to lose. Security. With CVEs (Common Vulnerabilities and Exposures) coming out as fast as they are, every component is another thing to patch, test, and redeploy. Six components means six upgrade treadmills. None of those costs show up in the demo. They all show up at 2 a.m. six months later.\nComplexity as job security # There\u0026rsquo;s a less innocent version of this, too. Some engineers make systems more complicated than they need to be on purpose.\nSometimes it\u0026rsquo;s about sounding smart. A system that\u0026rsquo;s hard to explain impresses non-technical coworkers and leaders. Sometimes it\u0026rsquo;s about job security. If you\u0026rsquo;re the only person who understands how everything fits together, the thinking goes, the company can\u0026rsquo;t afford to lose you.\nThat\u0026rsquo;s the Raptor 1 instinct turned into a career strategy. It optimizes for how the system looks, and for the person who built it, rather than for the people who depend on it.\nKeep it simple, for whoever comes next # I try to live by the KISS rule: Keep It Simple, Stupid. I want to build systems simple enough that anyone on the team can understand them, run them, and change them.\nPeople sometimes ask whether that makes me afraid of losing my job. Maybe a little. But if a company values complexity over stability and maintainability, it probably isn\u0026rsquo;t the right company for me. And honestly, I don\u0026rsquo;t want to be the only person who knows how something works. That\u0026rsquo;s the person who gets the call in the middle of the night, or on vacation.\nThere\u0026rsquo;s a more personal reason, too. I\u0026rsquo;m not a genius, and my memory isn\u0026rsquo;t great. When I come back to something a year from now, I want it to be easy for me to understand. A simple system is a gift to my future self, and to everyone else who has to work on it after me.\nHave you ever looked at code you wrote a few years ago and thought, why on earth did I do it that way? That\u0026rsquo;s common, and it\u0026rsquo;s a good thing. It means you\u0026rsquo;ve gotten better. If it ever stops happening, you\u0026rsquo;ve probably stopped growing.\nAdd complexity last, and write down why # My rule is to start as simple as possible and stay that way for as long as possible. Complexity is a last resort, added only when the simple version genuinely can\u0026rsquo;t do the job.\nWhen you do add it, document it. Not just what the system does, but why it\u0026rsquo;s built that way and what you tried first. The next person, or future you, needs to know which complexity is load-bearing and which is left over from a problem that no longer exists.\nAnd no, code is not self-documenting. Code can tell you what it does. It can\u0026rsquo;t tell you why, what alternatives you rejected, or what will break if you change it. You need real docs.\nThe time I deleted 90% of an app # Early in my career, shortly after starting a new job, I was handed an internal web app nobody liked. The person who wrote it had left the company. It was buggy, users complained constantly, and it had been passed around the team like a hot potato for years. Most people who touched it did the bare minimum and moved on.\nBefore writing a line of code, I spent a week getting to know it. I went through every open issue and read the code. Then I interviewed the users: what did they actually use the app for, which features mattered, and what was missing? It turned out to be a very simple app that didn\u0026rsquo;t need to do much.\nBy the end of that week I knew where the warts were. The biggest one: instead of using standard authentication and authorization libraries, the original author had written their own. Thousands of lines of code, full of bugs.\nI had a choice. Fix it, or rip it out? Going back to my notes, I realized much of that code wasn\u0026rsquo;t even being used. It was a half-built framework of dead code, written for a future that never came. So I took it out, and the app immediately got easier to understand. A few more conversations with users showed that some of the code that was being used wasn\u0026rsquo;t needed either. It was overkill.\nTwo weeks later, I had removed about 90% of the code. After a lot of testing, I closed almost every open bug. The next release wasn\u0026rsquo;t just more stable, it was much faster too. A couple of releases after that, the complaints stopped completely.\nThe app never needed a hero. It needed someone to take the time to understand it, then cut it down to the minimum its users actually needed. And if the original author hadn\u0026rsquo;t tried to future-proof it with features it would never use, it probably wouldn\u0026rsquo;t have ended up in that state at all. There\u0026rsquo;s a name for that lesson: YAGNI, or You Aren\u0026rsquo;t Gonna Need It.\nBut you still have to ship # I know this all sounds great in a blog post. In real life you have deadlines, customers waiting, and things that need to get out the door. Shipping matters too.\nSo ship something that isn\u0026rsquo;t perfect. That\u0026rsquo;s the great thing about software: we can keep improving it after it\u0026rsquo;s released. SpaceX didn\u0026rsquo;t wait for Raptor 3 before flying. Raptor 1 and 2 flew, and what they learned is what made Raptor 3 possible.\nThe key is to be honest about the rough edges. Write them down before you ship, then come back after the release and clean them up. Always leave the code a little cleaner than you found it. Over time, it gets there.\nStarting clean is always better. But that isn\u0026rsquo;t always an option, and we have to accept that and keep improving anyway.\nBuild the Raptor 3 # There\u0026rsquo;s an old line, usually credited to Blaise Pascal, apologizing for a long letter because he didn\u0026rsquo;t have time to write a short one. Simple takes more time, more understanding, and more discipline than complicated. It\u0026rsquo;s as much art as engineering.\nSo here\u0026rsquo;s my ask, for anyone building systems, products, or processes. Don\u0026rsquo;t stop at the version that works. Keep going until it\u0026rsquo;s the version that\u0026rsquo;s easy. Take on the complexity yourself so your users, your teammates, and future you don\u0026rsquo;t have to carry it.\nIt won\u0026rsquo;t look as impressive. Nobody will stare at it and marvel at all the pipes. But it\u0026rsquo;ll be lighter, stronger, and easier to live with. Just like Raptor 3.\nSources # Raptor 3 reveal and specs, NASASpaceflight, August 2024 (citing SpaceX) Header photo: SpaceX ","date":"November 10, 2026","externalUrl":null,"permalink":"/2026/11/10/simple-is-harder-than-complicated/","section":"Posts","summary":"","title":"Simple Is Harder Than Complicated: Lessons from the Raptor Engine","type":"post"},{"content":"","date":"November 10, 2026","externalUrl":null,"permalink":"/tags/simplicity/","section":"Tags","summary":"","title":"Simplicity","type":"tags"},{"content":"","date":"November 10, 2026","externalUrl":null,"permalink":"/tags/software-architecture/","section":"Tags","summary":"","title":"Software-Architecture","type":"tags"},{"content":"","date":"November 10, 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"September 1, 2026","externalUrl":null,"permalink":"/tags/argo-cd/","section":"Tags","summary":"","title":"Argo-Cd","type":"tags"},{"content":"","date":"September 1, 2026","externalUrl":null,"permalink":"/tags/book/","section":"Tags","summary":"","title":"Book","type":"tags"},{"content":"","date":"September 1, 2026","externalUrl":null,"permalink":"/tags/gitops/","section":"Tags","summary":"","title":"Gitops","type":"tags"},{"content":"Most teams I talk to can deploy to Kubernetes. That part is mostly solved. What they can\u0026rsquo;t easily answer is a simpler set of questions: What version is running in staging? How did it get there? Is it safe to promote to production?\nI call that the promotion gap, and it\u0026rsquo;s the reason I\u0026rsquo;m writing my second book, Kargo in Action, for Manning. It\u0026rsquo;s available today through the Manning Early Access Program (MEAP).\nWhy this book # I\u0026rsquo;m the Head of Engineering at Akuity, where we build a GitOps platform on Argo CD and Kargo. Argo CD is great at making a cluster match what\u0026rsquo;s in Git. But it doesn\u0026rsquo;t decide what should be in Git for each environment, or when a change should move from dev to staging to production. Teams fill that gap with CI scripts, Slack approvals, and a lot of tribal knowledge. Kargo was built to replace that duct tape with a real promotion engine.\nWriting with the creators of Kargo # I\u0026rsquo;m not writing this one alone. My co-authors are Jesse Suen and Kent Rancourt, the creators of Kargo. Jesse also co-created Argo CD. Writing with the people who designed the tool means the book can explain not only how Kargo works, but why it works that way.\nA different kind of book project # The last time I wrote a book, it was a sprint. I took over the Docker Cookbook, Second Edition partway through and had about two weeks to rewrite most of it before a family vacation. It was fun, but it was a whirlwind.\nKargo in Action is the opposite: a book planned from scratch, with co-authors, written chapter by chapter over months. We will keep adding new chapters each month until the book is complete. The great thing about MEAP is you can give us feedback as we are writing the book so we can go back and improve the chapters we released already to make them even better.\nWhat\u0026rsquo;s in the book # The book follows one company as it grows from a single-service pipeline into a platform that serves dozens of teams and services. Along the way you\u0026rsquo;ll learn how to:\nRun Kargo alongside Argo CD and do your first end-to-end promotion Design promotion pipelines across environments, release candidates, and hotfix paths Automate low-risk promotions, and gate production behind approvals and verification Roll back cleanly when a release goes sideways Produce the audit trail your compliance reviewers actually want If you run Argo CD today and your path to production still depends on scripts and manual steps, this book is for you.\nRead it now, in early access # With MEAP you get chapters as we write them, plus the finished book when it ships. It\u0026rsquo;s also the best way to shape the final book: early readers send feedback that goes straight into the next revision.\nGet Kargo in Action from Manning\nWatch the webinar # If you\u0026rsquo;d like a preview first, watch the Kargo in Action webinar:\nIf you pick up a copy, let me know what you think. I\u0026rsquo;d love to hear what\u0026rsquo;s working, what\u0026rsquo;s confusing, and what you want to see covered.\n","date":"September 1, 2026","externalUrl":null,"permalink":"/2026/09/01/kargo-in-action/","section":"Posts","summary":"","title":"I'm writing another book: Kargo in Action","type":"post"},{"content":"","date":"September 1, 2026","externalUrl":null,"permalink":"/tags/kargo/","section":"Tags","summary":"","title":"Kargo","type":"tags"},{"content":"","date":"September 1, 2026","externalUrl":null,"permalink":"/tags/kubernetes/","section":"Tags","summary":"","title":"Kubernetes","type":"tags"},{"content":"","date":"March 29, 2021","externalUrl":null,"permalink":"/tags/30-60-90/","section":"Tags","summary":"","title":"30-60-90","type":"tags"},{"content":"","date":"March 29, 2021","externalUrl":null,"permalink":"/tags/hiring/","section":"Tags","summary":"","title":"Hiring","type":"tags"},{"content":"","date":"March 29, 2021","externalUrl":null,"permalink":"/tags/managing/","section":"Tags","summary":"","title":"Managing","type":"tags"},{"content":"In the last blog post, we talked about ramping up your new developers as quickly as possible and we concentrated on their local development environment and why it is important to get them up and running quickly. Getting that setup is important but is only one step of many that you need to do to make sure they become a fully functional team member.\nAn employee will need to make sure they have system access, business knowledge, employee connections as well as the skills required for their job. Every new employee is going to be different when they start. Some are going to have some of the skills and knowledge, some might not have any. You need a way to determine what their gaps are and create a plan for that employee to make sure they know what they need to do to get to the point where they have everything they need to be a fully competent member of the team, able to take on tasks with little to no hand-holding.\nThat is where a 30-60-90 plan comes into play. Before an employee\u0026rsquo;s first day you should create a 30-60-90 plan tailored to that new employee so that they know what they need to do and learn on their first day. We want to set them up for success, and this will help us do that.\nBefore I even open up a new job requisition I like to create a rough 30-60-90 along with the job description so that I can list what I want this person to work on, and how they can ramp up. If you can’t describe this, maybe you don’t need to hire someone yet? There is nothing worse than hiring someone and then having nothing for them to do, and then having to lay them off.\nA 30-60-90 is a document that lays out an action plan for the employees\u0026rsquo; first 30, 60, and 90 days on the job. It is important to set your new employee up for success, and a 30-60-90 is the perfect way to do that. It lets them know what you expect from them and gives them a clear guide to follow in a normally chaotic time for them. It is also important to help each other see how they are progressing and to make sure they are not falling behind.\nThe goal is to onboard the new employee as quickly as possible so that they can be a fully functional member of the team with as little effort and impact on the rest of the team as possible. If you are noticing that they are not completing the items on time you can step in and try and correct it before it becomes a bigger issue. Some people treat the 90 days as a probationary period and if they pass they get off probation.\nIt is also helpful to determine if you hired the wrong person. If after 90 days they haven’t done what you would expect they should have done, you can put them on a Performance Improvement Plan (PIP) and extend the 90 days to 120 or 150 days, with clear goals, if they don’t complete them by that time, you will let them go.\nHiring is hard and sometimes you make mistakes, when that happens the goal is to recognize that as soon as possible so that you can correct it and get the right person for your team. If you don’t have a 30-60-90 it might be harder to notice and when you do notice it will be harder to put on a PIP and manage out. The 30-60-90 and a PIP go together quite nicely and if you use them correctly you can speed up the process of removing people that are not a good fit.\nThe 30-60-90 is a living document you use in your regular 1 on 1\u0026rsquo;s and if you both realize something doesn\u0026rsquo;t fit, feel free to move it around, remove or add something. Just make sure you are both in agreement with any change so that there is no confusion.\nTo make it easy to track an employee\u0026rsquo;s progress, I usually set up a meeting 30, 60, and 90 days from their state date for check-in meetings. This is important because it shows the employee that you are serious about these dates and it gives you a dedicated time to review the progress, and are great coaching opportunities to give feedback. These will not replace your normal 1:1’s and you should still be getting updates and coaching in those as well.\nTo make it easier to track their progress you can use a tool like Jira and setup epics for 30,60,90 and create tasks for each item in the 30,60,90 and assign them to the epics. Use these tasks when doing sprint planning so that the employees are part of normal sprint planning processes and stand-ups and it makes it easier for everyone to see the progress of those tasks by just looking at Jira.\nEven though it says 30-60-90 there is also a 0, -7, and a -14 which is dedicated to the hiring manager, those are the tasks that the hiring manager needs to do before the employee gets hired and on their first day. That includes any new employee HR intake forms, support tickets for system access, finding a mentor, connecting with the employee to make sure they have everything they need, giving them their login information, etc.\nI usually break it up into three parts learning, building, and owning? Each part is tied to the 30-60-90\nLearning: getting everything they need to get up and running, as well as meeting people, reading docs, and training. Building: working with others in the team mentor, buddy, etc, have them work with someone to show them the ropes Owning: when they are comfortable to have them start to do tasks on their own, and hopefully, by the end of the 90 days, they are a self-sufficient member of the team. Some people finish their 30-60-90 much quicker than others, that is fine to end early. It might be fine if they are a little late, but if they are very late that is a red flag that you need to take care of sooner than later.\nOnce your employee has finished the 90 days, do a debrief on the process and get feedback on how to make it better for the next person. Go back and update the docs before you forget\nAfter you do a few of these you will see patterns for what works and what doesn’t and hopefully they get better over time and it will be easier for you to write them\nEvery team and role is going to have a different 30-60-90 but there will probably be a few tasks towards the start that will be the same regardless, you can take that and make it a good starting template for when you need to create new ones.\nOnboarding new employees quickly and efficiently is critical to the success of any development team. Using a 30-60-90 action plan allowed me to make this process much more organized and also helped me determine if a recent hire might have been a mistake, and let me correct the problem before they became a larger problem. Using this process will hopefully help you as well.\n","date":"March 29, 2021","externalUrl":null,"permalink":"/2021/03/29/why-you-should-give-new-employees-a-30-60-90/","section":"Posts","summary":"","title":"Why you should give new employees a 30-60-90","type":"post"},{"content":"","date":"March 28, 2021","externalUrl":null,"permalink":"/tags/developers/","section":"Tags","summary":"","title":"Developers","type":"tags"},{"content":"","date":"March 28, 2021","externalUrl":null,"permalink":"/tags/on-boarding/","section":"Tags","summary":"","title":"On-Boarding","type":"tags"},{"content":"When you hire someone new for your development team, one of the primary goals is to ramp up that person so that they can be a fully contributing member as soon as possible.\nWhen dealing with developers one way to decrease this time is to make it as easy as possible for someone to commit code to production on their first day. For a lot of companies, this isn’t possible and I hear horror stories of new employees taking a week or two just to get their local environments configured and up and running.\nThis could be caused by several different reasons but it usually boils down to installing all of the dependencies that are required to run the development stack. This has gotten even worse over the years as more and more companies are embracing micro-services over monoliths. Now instead of just setting up one environment, you need to set up many. Luckily thanks to tools like Docker, Docker Desktop, and Docker Compose you can easily get all of your services running in containers, and once you have that you can easily script the setup.\nAn ideal new employee set up for their first day would look something like this.\nlog in to their new laptop Get access to the team source code repository (GitHub, etc) Clone the project to their laptop Run the bootstrap.sh script which will download and install dependencies on to their laptop Read the docs and start up the stack (docker-compose up) Find a minor task to work on. Run unit tests to make sure nothing broke. Make a minor change, commit code, submit Pull Request to get the code reviewed. If you can get this set up or something similar you are on the right track. Just think how much time and money you are saving by cutting down the Initial Time To commit (ITTC) for new people who join your project. It doesn’t work for just new employees, it could be important for crunch times when you need to pull in folks from other teams to help make a deadline.\nIf it takes a week to get someone set up before they could help, then it isn’t possible to easily move people around. That means you aren’t as flexible as you could be, and that will translate to either missed deadlines or making current team members work longer hours, which also isn’t ideal. Too many of those and you could burn out your developers.\nIf you have more than one project/team, you should try and have a similar process for all of them. This will make it easy for people to move around between teams and projects when needed and allows you to build common tooling to make it easier to maintain.\nAnother advantage to this is that it makes it easier to upgrade developer laptops and replace them if they break. If it takes a while to get the laptop just right you will find developers would rather hold on to an older laptop than upgrade and go through all the hassle of setting it up again. This means your team could be using slower, out-of-date hardware, which might take longer to build and run tests and thus slows the whole team down. If a laptop dies you want to be able to get that developer back up and running as quickly as possible. With the help of the cloud, and SaaS services you shouldn’t need much installed on local machines anymore, so the recovery times should be much quicker if you do things right.\nThe next time you add a new person to your team have them keep track of how long it takes to get up and running. If it isn\u0026rsquo;t under a day then it might make sense to dedicate some time to improve it. You don\u0026rsquo;t need to solve it all at once, just keep at it and keep improving, over time you should be able to get it down to a reasonable amount of time.\nNow that we have talked about the first day of a developer, my next blog post will go a little further and help you plan out the first 90 days .\n","date":"March 28, 2021","externalUrl":null,"permalink":"/2021/03/28/on-boarding-new-developers-to-the-team/","section":"Posts","summary":"","title":"On-boarding new developers to the team","type":"post"},{"content":"As an engineering manager, one of the things that you will need to do from time to time is hire new employees for your team. I have been involved in hiring developers for around 15 years now. I have done everything from just being on an interview panel, all the way to the hiring manager. Each of the roles on the interview panel has different responsibilities but they helped me finely tune the hiring practice I use today. It is set up to be fair and to hopefully eliminate any unconscious biases that might be in the process.\nFor all new job openings, I first write up a job description and a 30-60-90. I will go into 30-60-90\u0026rsquo;s in a future blog post. For the job description, we try and keep it as simple as possible and let people know what we are looking for. We break it down into three parts. The first part talks about the team and the role, and what we are looking for. We then include a list of minimum requirements (need to have), and then a list of nice to have.\nI try to keep the list of must-have requirements as small as possible and I also try not to require college degrees unless required. This is important to not exclude the non-traditional folks. Some of the best developers I have worked with didn’t have a computer science/engineering degree or didn’t even go to college. If they have the skills, I don’t care how they acquired them.\nNow that we have the job description and 30-60-90 we post the job, and we wait for resumes, but we also scan our networks to see if there is anyone that we know that would be a good fit. The job market is so hot right now, if you don\u0026rsquo;t go looking it will be hard for people to fill your roles.\nOnce we get some candidates we do a quick 30-minute phone screen with the recruiter to filter out anyone that doesn’t fit. We gather their salary requirements, and other helpful information as well as describe the position to them, to make sure it is something they are interested in pursuing.\nIf they pass the recruiter\u0026rsquo;s phone screen we use an online service to send them a take-home self-assessment, and a small coding challenge. We give the same assessment to everyone regardless of the level and experience. The goal of the assessment isn\u0026rsquo;t to eliminate anyone, it is to help us gather information upfront so that we can ask fewer questions during the interviews.\nThe assessment is broken up into three parts\nA list of tech skills where people self-rate themselves from 0-10. 0 they never heard of it, 10 they invented it, or wrote a book. A few open-ended question, like what are you goals for the next few years, etc. A few entry-level coding exercises. write a function to do blah, that sort of thing. They can take as much time as they need and they can pick whatever language they want. We take their answers from the assessment, resume, and recruiter notes and give them to the hiring manager. If the hiring manager is happy they will set up a hiring manager phone screen. During the phone screen, the hiring manager goes over their assessment and their resume, and describes the position in a little more detail, and lets the candidate ask as many questions as they need. The manager also gives more background on the company and how their team fits in the grand scheme of the company. At the end of the phone screen, if both the candidate and the hiring manager are interested in continuing, they move to the next round.\nThe next round consists of two team interviews. All of the questions for these interviews were determined beforehand and signed off by HR to make sure there are no issues with them. All candidates get the same set of questions to make sure no one is given an advantage. The interviews consist of two people from the team and they last around 30-45 minutes each.\nThe last interview is a technical interview that takes 60 minutes and consists of two parts. In the first part, the candidate is given a scenario and is asked to peer with the interviewer to solve the problem. The candidate leads the problem solving with the interviewer there to answer questions and help move it along if they get stuck.\nThe second part is a peer programming exercise where the candidate is given access to a web-based IDE and given a few questions they need to solve. They can pick the language of their choice to solve the challenge. The interviewer is there it answers any questions and helps guide the candidate if they get stuck. The interview ends with time for the candidate to ask any questions they might have.\nThe last part of the process is a brief debrief meeting between the recruiter and anyone on the interview panel. They discuss the pros and cons of the candidate and then decide if they recommend hiring them or not.\nCandidates are usually part of a cohort and we interview everyone in that cohort before we decide who to hire. After every debrief we rank the candidates we have interviewed so far and keep a running list. When we have interviewed everyone we make an offer to the person on the top of the list. If someone rejects the offer, we move to the next on the list. If we get to the bottom of the list, then we start a new cohort and do the whole process all over again.\nWe take our time when it comes to hiring, it is better to make sure you have the right person than accidentally hire someone that isn’t a good fit.\nHiring the wrong person can take a lot of effort to train and then if needed manage the person out. One wrong hire could set you back 6 months by the time you recognize your mistake, manage them out, open up a new job, hire and then train someone else. So an extra day or two, or a few more questions is well worth it.\nIn summary here are the different steps in our hiring process.\nCreating the Job Description and 30-60-90, and posting it on the different job boards Start the search to find good candidates and filter through the resumes that get submitted Recruiter phone screen Candidate completes the take home self-assessment Hiring manager phone screen two team interviews technical interview Debrief meeting Make an offer In my next blog posts, I\u0026rsquo;ll talk about the 30-60-90 and what to do once you hire someone.\n","date":"March 27, 2021","externalUrl":null,"permalink":"/2021/03/27/how-i-hire-software-developers/","section":"Posts","summary":"","title":"How I hire software developers","type":"post"},{"content":"When Apple introduced MacOS Catalina, it came with some security features to make sure you were only using trusted binaries. This requires that the binaries are signed and notarized by Apple themselves, or else you get an error.\nMacOs Catalina error for unsigned binary There are ways around this error, but that makes it difficult if you want distribute your binaries to a large group of people. It could slow down adoption.\nAt work, I wrote a Golang binary that makes it easy for people to get temporary AWS API keys using their work Okta login. I cross compile the binary so that Windows, Linux and MacOS users can all use it.\nIt has worked great with very limited issues until teams started upgrading to MacOS Catalina. Once they upgraded, it would give them an error saying the binary they were using was not trusted because it wasn’t signed.\nTo solved this issue, I did some research and got the binaries signed. But this was a manual process, and because I’m lazy, I wanted to automate this step so that I didn’t have to do this by hand every time a new release came out.\nTo automate this process I used GitHub actions, and it worked really well, so I’m going to share with you how I did it, with a simple example golang app.\nThis requires that you have an Apple Developer account, so if you don’t have one yet, you will need to get one.\nGetting started # In order to show you how this works, I created a simple golang app and put all my code in a github repo. Feel free to check it out to see the end result.\nTo help with the building of the Golang binary I\u0026rsquo;m using a Makefile and a few helper scripts and doing the build in a Docker container to keep everything nice and reproducible. The other benefit of doing it this way, is that you can build the binaries like normal on your local machine, and use the same code to build in the github action. If you did all building in github actions you might have to have two different ways to build your code, and that could lead to drifting issues in the future if the build process changed at all.\nThere are a few scripts that I created to make everything easier:\nbuild.sh which will create a docker container and call build_app.sh which will call make build and that will compile and create three different binaries one for Mac, Linux, and Windows, and put them in a /release directory. sign.sh This is a wrapper script that will call make sign which will kick off the binary signing process for MacOS using Gon. Gon # To make the signing process much easier I\u0026rsquo;m using Gon written by Mitchell Hashimoto who is the founder at Hashicorp (Terraform, Vagrant, etc). Gon is what they use to sign Terraform, etc. If it is good enough for Terraform, it is good enough for me.\nGon Prerequisite: Acquiring a Developer ID Certificate # Before using gon, you must acquire an Apple Developer ID Certificate. To do this, you can either do it via the web or via Xcode locally on a Mac. Using Xcode is easier if you already have it installed.\nVia the web:\nSign into developer.apple.com with valid Apple ID credentials. You may need to sign up for an Apple developer account.\nNavigate to the certificates page.\nClick the \u0026ldquo;+\u0026rdquo; icon, select \u0026ldquo;Developer ID Application\u0026rdquo; and follow the steps.\nAfter downloading the certificate, double-click to import it into your keychain. If you\u0026rsquo;re building on a CI machine, every CI machine must have this certificate in their keychain.\nVia Xcode:\nOpen Xcode and go to Xcode =\u0026gt; Preferences =\u0026gt; Accounts\nClick the \u0026ldquo;+\u0026rdquo; in the bottom left and add your Apple ID if you haven\u0026rsquo;t already.\nSelect your Apple account and click \u0026ldquo;Manage Certificates\u0026rdquo; in the bottom right corner.\nClick \u0026ldquo;+\u0026rdquo; in the bottom left corner and click \u0026ldquo;Developer ID Application\u0026rdquo;.\nRight-click the newly created cert in the list, click \u0026ldquo;export\u0026rdquo; and export the file as a p12-formatted certificate. Save this somewhere. You\u0026rsquo;ll never be able to download it again.\nTo verify you did this correctly, you can inspect your keychain:\n$ security find-identity -v 1) 4194587FE60D93D416CF3F4669FF913C7BBA4271 \u0026#34;Developer ID Application: Your Name (GK80BB2A7)\u0026#34; 1 valid identities found More notes on getting an Apple Developer Certificate # In order to get an Apple developer certificate I did the following.\nCreated the CSR using the Mac keychain app, and stored it locally on my machine. How to create a CSR: In order to create a developer certificate you need to be an account holder. If you are not an account holder, you will need to find out who is, and ask them to help. I sent them by CSR, but I\u0026rsquo;m not sure if that is actually needed or not, but figured I would leave that step here for future Ken incase I ever need it. In order to sign and notarize you need to have an app specific password (regular passwords that require 2FA will fail with the error \u0026ldquo;We are unable to create an authentication session. (-22016)\u0026rdquo;) Take your Apple Developer username and App Specific password and add them as a secret in your Github repo, and if you want to do signing and notarizing locally, you will also need to set the password in your ENV under AC_PASSWORD. AC_USERNAME AC_PASSWORD Exporting the Developer Certificate # Installing the developer certificate on your local machine is good for local builds, but we want to do this in a github action, so we need to get the certificate in a format that will allow us to import it in our build machine.\nExport the Certificate from the keychain more details here Click on \u0026ldquo;My Certificates\u0026rdquo; on left side, select the \u0026ldquo;Developer ID Application: \u0026hellip;\u0026rdquo; Certificate Click on arrow next to the name so that it shows the private key as well. Highlight both rows, and right click \u0026ldquo;export 2 items\u0026rdquo;. Give it a Name (Certificate.p12), and then a good password. Your new p12 certificate will be in the location where you specified. Keep track of that password, it is needed for the github action (as a secret) Copy the certificate in base64 format, and store as a github secret on the repo cat Certificate.p12 | base64 | pbcopy Go into your github repo and set the github secrets for the certificate and the password APPLE_DEVELOPER_CERTIFICATE_P12_BASE64 APPLE_DEVELOPER_CERTIFICATE_PASSWORD Now that you have the certificate and password as a github secret, the github action can use them to build and sign the binaries.\nGon Configuration # Here is the configuration we are going to use. For details on the config, go read the gon documentation This depends on two ENV variables. AC_USERNAME and AC_PASSWORD.\nNote: If you try and use @env:AC_USERNAME in the apple_id section it won\u0026rsquo;t work. Either hard code it, or leave it out and add as an ENV variable, the @env function doesn\u0026rsquo;t work for username.\n{ \u0026#34;source\u0026#34; : [\u0026#34;./release/darwin/hello-github\u0026#34;], \u0026#34;bundle_id\u0026#34; : \u0026#34;com.kencochrane.hello-github\u0026#34;, \u0026#34;apple_id\u0026#34;: { \u0026#34;password\u0026#34;: \u0026#34;@env:AC_PASSWORD\u0026#34; }, \u0026#34;sign\u0026#34; :{ \u0026#34;application_identity\u0026#34; : \u0026#34;4194587FE60D93D416CF3F4669FF913C7BBA4271\u0026#34; }, \u0026#34;dmg\u0026#34; :{ \u0026#34;output_path\u0026#34;: \u0026#34;./release/darwin/hello-github.dmg\u0026#34;, \u0026#34;volume_name\u0026#34;: \u0026#34;hello-github\u0026#34; }, \u0026#34;zip\u0026#34; :{ \u0026#34;output_path\u0026#34; : \u0026#34;./release/darwin/hello-github.zip\u0026#34; } } Github Action # Now that we have everything in place, we can create a github action that will trigger when we push a new tag to our repo. That will build our binaries on a Linux Runner, and then sign and notarize on a MacOS Runner. I am building on Linux and signing on MacOS vs doing it all on MacOS because the MacOS Runner doesn\u0026rsquo;t come with Docker installed and because of this you can\u0026rsquo;t run Docker containers. It also lets me show you how to use the upload and download artifact feature that might become useful to you sometime in the future.\nBuild # Here is our github action workflow file. The first build step is pretty simple, it will checkout the code, and then run the build.sh script to build the binaries, then it will upload the artifacts so that they are available in the next step. It does this in an ubuntu-latest hosted runner.\nbuild: name: Build the binaries runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v2 - name: Build run: | ./build.sh - name: upload binary artifacts uses: actions/upload-artifact@v2 with: name: binaries path: release/ Release # This step is a little more complicated, it will run on the macos-latest hosted runner.\nIt does the following.\ncheckout the code download compiled binaries from the previous step Import the developer certificates from the github secrets Install gon via homebrew signs the mac binaries using gon, referencing the ENV variables for username and password. Zip up the release directory with all the binaries create a github release attach the release binaries to the github release, so they are all together. If successful you will have a new github release with your signed and notarized binaries when ever you tag your code and push to github.\nrelease: needs: build runs-on: macos-latest steps: - name: Checkout uses: actions/checkout@v2 - name: Download-Binaries uses: actions/download-artifact@v2 with: name: binaries path: release/ - name: Import Code-Signing Certificates uses: Apple-Actions/import-codesign-certs@v1 with: # The certificates in a PKCS12 file encoded as a base64 string p12-file-base64: ${{ secrets.APPLE_DEVELOPER_CERTIFICATE_P12_BASE64 }} # The password used to import the PKCS12 file. p12-password: ${{ secrets.APPLE_DEVELOPER_CERTIFICATE_PASSWORD }} - name: Install gon via HomeBrew for code signing and app notarization run: | brew tap mitchellh/gon brew install mitchellh/gon/gon - name: Sign the mac binaries with Gon env: AC_USERNAME: ${{ secrets.AC_USERNAME }} AC_PASSWORD: ${{ secrets.AC_PASSWORD }} run: | ./sign.sh - name: Zip up release directory run: | zip -r release-binaries.zip release/ - name: Create Release id: create_release uses: actions/create-release@v1 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: tag_name: ${{ github.ref }} release_name: Release ${{ github.ref }} draft: false prerelease: false - name: Upload release binaries Zip id: upload-release-asset uses: actions/upload-release-asset@v1 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: upload_url: ${{ steps.create_release.outputs.upload_url }} asset_path: release-binaries.zip asset_name: release-binaries.zip asset_content_type: application/zip Conclusion # Hopefully this blog post along with the accompanying GitHub repo will make it easier for you to use Github Actions to build your Golang binaries and then sign and notarize them for MacOS, so that your users don\u0026rsquo;t get any security warnings.\nIf this is a topic that you find interesting and you are looking for a job, please reach out, I\u0026rsquo;m always looking to hire smart people who want to solve interesting problems. We hire fully remote folks, so if you don\u0026rsquo;t live near one of our offices around the world, you can work from home, no problem with us.\n","date":"August 1, 2020","externalUrl":null,"permalink":"/2020/08/01/build-and-sign-golang-binaries-for-macos-with-github-actions/","section":"Posts","summary":"","title":"Build, notarize, and sign Golang binaries for MacOS with GitHub Actions","type":"post"},{"content":"","date":"August 1, 2020","externalUrl":null,"permalink":"/tags/github/","section":"Tags","summary":"","title":"Github","type":"tags"},{"content":"","date":"August 1, 2020","externalUrl":null,"permalink":"/tags/github-actions/","section":"Tags","summary":"","title":"Github-Actions","type":"tags"},{"content":"","date":"August 1, 2020","externalUrl":null,"permalink":"/tags/golang/","section":"Tags","summary":"","title":"Golang","type":"tags"},{"content":"","date":"August 1, 2020","externalUrl":null,"permalink":"/tags/macos/","section":"Tags","summary":"","title":"Macos","type":"tags"},{"content":"","date":"August 1, 2020","externalUrl":null,"permalink":"/tags/signing/","section":"Tags","summary":"","title":"Signing","type":"tags"},{"content":"","date":"September 24, 2019","externalUrl":null,"permalink":"/tags/bucket-list/","section":"Tags","summary":"","title":"Bucket-List","type":"tags"},{"content":"","date":"September 24, 2019","externalUrl":null,"permalink":"/tags/docker/","section":"Tags","summary":"","title":"Docker","type":"tags"},{"content":"One of the things that I have always wanted to do was write a book. When I was at Docker I had a few offers from publishers to write a book, and I even started writing one called the Docker guidebook and you can find it on Github. Since Docker was growing so fast and I was spending so much time working that I didn\u0026rsquo;t have a lot of free time to write a book. I decided instead to take my free time and spend it with my family. That was a pretty easy decision, and I would make it again if I had to with no regrets.\nMy dream of writing a book was still there, so when I had another opportunity to do it, I decided I would do what it would take to accomplish my goal. In July of 2019, I was approached by Packt Publishing to help them with a problem they had. They had a second edition of a book they were trying to finish up and the author that was hired to work on it wasn\u0026rsquo;t able to dedicate the time to finish it. They wanted to know if I could take over and finish the book. The problem was that the book was already behind schedule, and they wanted to get the whole thing written and published in the next 30 days. That was a tall task, but to make it even more difficult, I was planning on going on vacation with my family for 2 weeks of that time. That meant that if I accepted that challenge that I would need to do everything in the next 2 weeks. I talked it over with my wife and we decided to go for it.\nOnce I had the contract signed, for the next two weeks I woke up really early before work, and then again after I got home from work as well as on the weekends to rewrite most of the existing chapters and also write a brand new chapter from scratch. I had never written a book before so I was learning as I was going, and even though it was a lot of work, it was fun. The most annoying part of writing the book was creating the code examples with screenshots. It was hard getting them to look consistent, and making sure they are 100% accurate. As I finished a chapter I was able to submit it to the editors via their web portal, and since they were located in India I would usually have the feedback in my queue the next morning. You can tell Packt has done this before because they have a very efficient process. It isn\u0026rsquo;t perfect but it seems like they committed to working at it until it gets there.\nI was able to successfully finish all of my chapters to the editors before I went on vacation, and I still took my laptop with me so that I could handle any issues that might pop up while I was gone. Luckily most of the things that popped up were pretty simple and I only had to work a few hours the whole vacation, and I normally did it early in the morning before anyone else woke up, so it didn\u0026rsquo;t interfere with any of our plans. After a little while, the edit requests stopped coming in and then everything went silent. Out of nowhere, I got an email congratulating me that my final draft was accepted and it was finished. It went on sale on Amazon the next day, and with that my whirlwind month of writing a book was finished. I was now a published author, and it felt good.\nIf you are interested in buying a copy it is available in print or ebook at a number of places including Amazon.\nDocker Cookbook Second Edition at Amazon Docker Cookbook Second Edition at Packet Some people have asked me if I would write a book again, and I think I would if it was the right project. You don\u0026rsquo;t make a lot of money writing books, so I would really need to love the topic and feel that it would be helping people before I commit the time and effort.\nDocker Cookbook Table of Contents # Introduction and Installation Working with Docker Containers Working with Docker Images Network and Data Management for Containers Docker Use Cases Docker APIs and SDKs Docker Performance Docker Orchestration and Hosting a Platform Docker Security Getting Help and Tips and Tricks Docker on the Cloud Update March 2020 # I wanted to give you an update on how the book is doing. If you want to get rich then writing a book isn\u0026rsquo;t for you. Very few authors make enough money selling books to do it full time. Most people are like me, are doing it in their spare time. I didn\u0026rsquo;t want to write a book for the money, I did it to cross off an item on my bucket list, and the money was just an added bonus. It is also good for the resume, not many people can say they are a published author on a given topic. If you are applying for a job and it comes down between you and someone else, and they see that you wrote a book on a topic related to an area they are interested in, it might be enough to have them pick you. On its own, it probably won\u0026rsquo;t win you the job, but it is a nice bonus.\nI still haven\u0026rsquo;t sold enough books to cover my advance yet, but I\u0026rsquo;m hoping the next quarter will change that. If you want to help, please buy a book, it is still very relevant, and hasn\u0026rsquo;t gotten out of date yet :)\n","date":"September 24, 2019","externalUrl":null,"permalink":"/2019/09/24/i-wrote-a-book-docker-cookbook-second-edition/","section":"Posts","summary":"","title":"I wrote a book : Docker Cookbook Second Edition","type":"post"},{"content":"","date":"September 17, 2017","externalUrl":null,"permalink":"/tags/cashstar/","section":"Tags","summary":"","title":"Cashstar","type":"tags"},{"content":"On August 30th, 2017 BlackHawk Network acquired CashStar for $175 Million dollars in cash. This is good news for everyone involved, the CashStar employees, share holders, and for black hawk. This is a great ending, but how did they get where they are today, and how did it all start? I was lucky enough to be one of the first employees at CashStar, so I have some first hand knowledge of the early years. I have been meaning to write about this for a while, and with the recent sale, it looks like this is a good time to sit down and write it down, before I forget.\nHow it all started # Most people don\u0026rsquo;t know, that the CashStar that they know today, wasn\u0026rsquo;t the first CashStar. The first CashStar was originally based in California and was incubated at Coupons inc, and there was a separate CashStar Australia that was doing coupons in Australia and the U.K. The company had investors and funding, but it didn\u0026rsquo;t make it. It wasn\u0026rsquo;t doing well, so they decided to close up shop and return the rest of the money to the investors. I don\u0026rsquo;t know too much about this first company, if you know anything, please let me know and I\u0026rsquo;ll update the post.\nCashStar the company still had $4 million dollars, but it had no employees or a product. The investors decided to reboot the company with a new founder and product. Steven Boal talked with David Stone about his idea, and the two decided to move forward with a company based on a new product called Virtual Gift Cards (VGCs). David would be CEO and run the company in Maine.\nDavid started in December 2007, and by February 2008 he had hired David Wood as the first employee, and Glenn Snyder as the VP of Engineering. Glenn was a friend of David\u0026rsquo;s and agreed to come on board as a contractor to help him hire and build the engineering team. Glenn\u0026rsquo;s first hire was Phelps Peeler as the director of backend engineering. Phelps started in June, and worked on the gift card authorizer that CashStar would use to authorize it\u0026rsquo;s gift cards.\nI had previously worked with Phelps at Wright Express, and a few weeks after he started at CashStar we had lunch at Ricettas, a pizza place around the corner from his new office. He was telling me about all of the cool things they were doing, and mentioned that they needed someone to build out the web side of the business, so they can take orders online. At this point in my career, I had many years of experience building out web applications including ones with an e-commerce component. It interested me enough that he convinced me to interview for a job. A couple of weeks later I was meeting with Glenn and David and they offered me a job. I accepted the job offer and I gave my notice, and started in August 2008.\nOne of my requirements for accepting the job at CashStar was that I wanted an Apple MacBook Pro. They agreed to this, and on my first day when Phelps and David saw that I had a MacBook, they were a little jealous. It was only a matter of time, before they also switched over to using a MacBook. When we started hiring people, the fact that we allowed them to pick which laptop they wanted, was seen as a big plus for us. I\u0026rsquo;m not sure if it single handedly convinced people to accept a job, but I\u0026rsquo;m sure it was one of the factors.\nIn August 2008 we also hired Phil Moran our first sales person and Laura Cannon our first client services / marketing person. Miklos Van Halen was also hired to help Phelps with the backend Java service. Now that we had enough people, we could get started building CashStar.\nThe Castle # The Castle our first office Our first real office was in an office building they called \u0026ldquo;The Castle\u0026rdquo;. The building used to be a youth detention center and then a while ago they remodeled it and turned it into offices. We were subletting our office from another company and it was pretty small. The office was less than 800 square feet, with access to a shared kitchen with a fridge, microwave, coffee machine and a table.\nWhen you walked in, there was a small conference room table on the left. Dave Wood had his desk, and then there were a few tables built into the walls, enough for 4-6 people to work. There were two offices, the one on the left was David Stone\u0026rsquo;s, and Phelps and I shared the one on the right. This was a good office size for about a year, and then we started to outgrow it, and we expanded into other office spaces on the same floor.\nOne of the great things about the Castle, was this big open field right next to it. On nice days, when we needed a break, we would go out and play kickball or bocce. It was a good way to relieve some stress.\nIn the kitchen we had our own fridge, and like all startups, we stocked it will some adult beverages.\nWe eventually outgrew this space, and moved to a much larger office on middle street in the old port section of Portland.\nBuilding CashStar # When I started working at CashStar, I was told I had six weeks to get my web application up and running in order for us to sign a customer and get them live in time for the all important Christmas shopping season. If we missed this deadline, we would lose our chance at making money that year.\nThe first thing I did, on my first day, was take an inventory for what what they currently had. There was Phelps and Miklos\u0026rsquo;s backend code for handling gift card authorizations written in Java. A badly broken .Net demo application, written by a consulting company. The website was a single static page hosted on a shared hosting service. We had no servers, or infrastructure of any kind. This was truly greenfield development, and it\u0026rsquo;s what I liked most about the project. There was no legacy code or production customers to hold us back, we could do what ever we needed in order to make our deadline.\nThe way I saw it, I had two options:\nKeep the .Net application and the consulting company who wrote it, and help them fix all the bugs and get it production ready. Throw everything away and start from scratch. After talking it over with Phelps and Glenn, I decided to pick option two, for the following reasons.\nI didn\u0026rsquo;t know .Net, and if we decided on going this route, I knew that we would stick with .Net going forward. That point forward we would be a windows shop. If I started from scratch, I would be able to build the site from the ground up, using the technologies that I wanted, and was the most comfortable with, and I would know everything about it. This was important incase I need to fix a bug in the middle of the night. There would be no learning curve, or dependencies on an external vendor. The demo application was only partially completed and it had lots of bugs, and it would have needed a lot of work to get it production ready. By throwing it out, we wouldn\u0026rsquo;t be losing much. By removing the external consulting company, we saved money on development costs, and as a startup every dollar counts. Once I decided to start over, we fired the consulting company, and I spent the next day, picking out the tech stack.\nTech stack # Over the years people have asked me a few times why I picked the tech stack that I did. It has been a while, but here are the reasons to the best of my memory.\nAt that point, I had 10 years experience building Java web application, and it was long enough time to know that Java and it\u0026rsquo;s many web frameworks were not great at building web apps, so I decided to use something else. After looking at all of the different options available (java, php, perl, python, ruby, etc) I narrowed it down to Python with the Django framework and Ruby with Ruby on Rails. I then went through the pros and cons of each language and framework, and I finally decided on using Python with Django, for the following reasons:\nI had used Python/Django for about a year at that point and had built a few web apps using it, and it was really nice. I was able to build things really quickly, and the code was really clean and easy to read. At that point in time Ruby on Rails 2.0 was released, but there wasn\u0026rsquo;t many books or gems that supported it. I didn\u0026rsquo;t want to write something in Rails 1.x when I knew I would need to upgrade to 2.x in the near future. Django had a really nice templating engine that allowed me to handle the customer branding the way I wanted, Ruby didn\u0026rsquo;t offer a comparable templating system at the time. Django\u0026rsquo;s documentation was really good, and it was fairly easy for developers to pick it up. One gotcha for Django was that it wasn\u0026rsquo;t 1.0 yet, but they were on track to be 1.0 before we were going to be live in production. So I developed on Django\u0026rsquo;s master branch, and updated every day, and fixed any issues that came up. Eventually they released 1.0, right before we went live, and it was pretty stable after that. Developing off of the ever changing Django master branch was a challenge. Whenever you would update to use the latest changes, you had to spend time trying to figure out what was going to break. Targeting 1.0 was the right move due to some of the features that were included in that version.\nWe made a few decisions on the tech stack, mostly because we knew the tools, and we didn\u0026rsquo;t have time to learn something new, and time was the utmost importance. We decided to use MySql over PostgreSQL, and subversion instead of git for these reasons. We picked Mysql because I had more experience administering MySql compared to PostgreSQL. Git was really new, and we didn\u0026rsquo;t want to risk losing time or code with a new versioning system.\nNow that I had the tech stack decided, I needed to build the application. I\u0026rsquo;m not really good when it comes to front end (css, UI/UX, etc), so to help me with that, we hired Brian McLaughlin as a part time contractor. I had worked with Brian previously so we were able to hit the ground running. He designed the pages, and I wrote all the code to make them work. After a few weeks we had a fully functional demo, that we were able to show to people and do some usability testing.\nNow that we had a functioning demo, I needed to work on getting everything working for real. I needed to buy some servers to host our application. If I was doing this today, I would normally spin up servers in Amazon Web Services (AWS), but back then, AWS wasn\u0026rsquo;t what it is today, and there was questions on whether it was PCI compliant or not. So instead I found a data center company that leased us five physical linux servers, a firewall and a hardware load balancer. There were two production servers connected to the load balancer, one dev server, one database server, and one backend server.\nOne mistake that I made when I bought the servers was that I didn\u0026rsquo;t realize that they had CPanel installed. CPanel is a web hosting control panel. It is suppose to make running a linux server easier if you don\u0026rsquo;t know what you are doing. If you know what you are doing, it is a real pain in the butt because it expects things a certain way, and if you change those things, it breaks. I spent a couple of sleepless nights trying to configure those servers the way we needed. This could have been prevented if I had paid more attention to the server order form. Lesson learned I guess.\nAfter a long six weeks, we were able to finish the first version of our gift card platform. It was very much an MVP (minimal viable product) but it worked. The front end would serve up a website where the customer would fill out an order form and enter their credit card information. It would connect to Authorize.net for credit card authorization and if it succeeded, it would connect to the backend system to get a gift card number. The front end would then build out the digital gift card, and send emails to the buyer (receipt) and the recipient (gift card). Now that we had a product to sell, we just needed clients.\nThe chicken and the egg # Once we had the product finished, we needed to sign customers. We had a hard time finding our first customer, it wasn\u0026rsquo;t because they didn\u0026rsquo;t like the solution, they just didn\u0026rsquo;t want to be first. They wanted to see someone else using it, to prove it wasn\u0026rsquo;t vaporware. We had a chicken and the egg problem, we had people who wanted to sign, but they didn\u0026rsquo;t want to be first.\nTo get around this, we faked it until we made it. Jeff Dixon was consulting with us, and he was friends with the owner of Gorham Bike and Ski (GBS). Gorham bike and ski was a local bike and ski dealer in the Portland area with three stores. We struck a deal with the owner to use our gift cards at no cost, in exchange for being our first customer, and we also agreed to give him free marketing to promote his stores.\nAfter we signed the deal, we had to actually get them setup. Since this was our first customer, we had no process for on boarding them to our platform. We had to figure everything out on the fly. We made a custom branded website that matched what their look and feel. This was important, because we were hosting those pages, and we didn\u0026rsquo;t want people to know they were leaving the other website to make a sale, this helped people trust us enough to make the purchase. The only way they would know they left the website, was to look at the URL in the browser.\nWe also needed to integrate with the POS\u0026rsquo;s at the three GBS stores. This allowed the people who were making purchases with their gift cards to scan the gift card. This was a real pain in the butt, and it required a lot of little hacks to get it up and running. After a stressful couple of weeks, we finally got it up and running and were able to do a real life test. We drove over to one of their stores and we bought some stuff with our new gift card. When it worked, we celebrated.\nDavid first GBS purchase With GBS now online we were able to let some of our potential customers know, so they could try it out. Now that it was no longer vaporware, we quickly signed Unos Pizzeria and Travelocity. Now the challenge was getting them live before the holiday rush.\nBumps in the road # One of the problems you have when you are trying to be the first person to do something, is that there is very little help when something goes wrong. It is always great to be first, but most people don\u0026rsquo;t understand how painful the path to getting there was.\nWe were building the first online gift card platform. People could go to our website, enter in their credit card, and send the gift card to someone over email. This sounds easy, but there is a lot to it.\nFirst we needed to find a credit card processor that would allow us to be their customer. Because of the high risk of fraud a lot of the companies didn\u0026rsquo;t want to touch us. You have to remember this was before Stripe, and the other credit card processors of today. We finally found a company that took a chance on it, and it wasn\u0026rsquo;t until after David called in a lot of favors.\nOnce we had the processor in place, we needed to connect to Authorize.net and also make sure our site was secure and PCI complaint. This was before tokenization services existed, so the credit card numbers actually hit our servers and that meant we had a higher level of PCI we needed to deal with. That meant audits, server scanning, and a lot of paperwork.\nEven after we got the credit card number, we don\u0026rsquo;t even know if that credit card number was stolen or not. Since everything is online, and we are sending gift cards via email, it make it easy for someone to get a list of stolen credit cards, place a bunch of orders and then check their email for the gift cards. They could then sell those gift cards, or use them to buy something. This put us at risk for both credit card fraud and money laundering.\nIf we accept the credit card transition, then we need to deliver the gift card via email. This sounds easy as well, but you wouldn\u0026rsquo;t believe how much trouble we had sending emails that didn\u0026rsquo;t get marked as SPAM. It took us a while, but we finally built up an reputation, and fixed the issues that were getting us marked as SPAM.\nIf everything worked perfectly and the recipient got their gift card in their email, they still needed to use it. They needed to print it out, and so we needed to make sure that our gift cards printed out correctly on all types of computers, and printers. Once again, it seems easy, but it is surprising, how much work went into making sure the gift cards looked good when they were printed out. If it didn\u0026rsquo;t fit on one page, it would waste paper. If they were blurry, then the barcode might not scan. If they printed in black and white vs color, does it still look good, and is it readable/usable?\nIf they were able to print it out and bring it to the store, does the clerk at the store know what to do? Are the barcode scanners at the POS able to scan the barcode? At the beginning, a lot of times it was a no, to both of these questions. With anything new, you had to train people how to use it, and that is a lot of work. If you do it wrong you will frustrate customers and they will not use it again. As you can see, there is a lot more to this problem then most people realize, and if it wasn\u0026rsquo;t for the great team that we had, we wouldn\u0026rsquo;t have made it.\nThe Pivot # When we first started building CashStar the goal was to build an open loop gift card network. Open loop is basically what you have with a prepaid Visa debit card. Anywhere that Visa is accepted, you can use the debit card. We wanted to do the same thing for CashStar gift cards. They could be branded for a particular company, or you could use our generic CashStar gift card, and use it at any of the companies that uses our gift card system.\nThis was a nice goal to have, but it was a lot harder to get up and running then we imagined. In order for you to have your gift cards accepted at a business, you need to integrate your gift card authorizer into that business\u0026rsquo;s Point of Sale (POS) systems. This means touching every POS for a business, and in some cases businesses had different POS versions. The number of POS systems on the market is fairly large, and it would require custom code and certifications for each one. We bought a POS and had it in our office to make it easier for us to test our system. Companies don\u0026rsquo;t make changes to these systems that often, because they are a pain to deal with. When we tried to sell to these companies, this was a non-starter for a lot of them.\nOn top of the POS problem, most companies already had gift card systems in place, and they didn\u0026rsquo;t want to have two separate gift card systems. One for physical cards, and one for digital cards.\nGiven those two barriers to entry, we decided to change our goal, and instead of providing an open loop gift card, we would instead integrate with their existing gift card system. This approach had it\u0026rsquo;s pros and cons. The pros were that we didn\u0026rsquo;t have to integrate with their POS systems, and there would only be one gift card system for both digital and physical gift cards. The con, was that we now needed to integrate with a 3rd party to use their gift card system.\nIntegrating with a gift card system for our client introduced another set of issues we needed to overcome. In many cases we were the first company to request this access from these gift card processors. Many of the companies didn\u0026rsquo;t have any policies in place for giving a 3rd party access to their clients account. We had to do a lot of work to get access to these systems, and to gain their trust. This included certifications and audits, to make sure we knew what we were doing. This makes sense, if we had a bug in our code we could cost our clients lots of money. The good news was that once we had one customer on a processor, it was a lot easier to get another, since we already had a relationship.\nIn order to handle these processor integrations we converted the backend code from a gift card authorizer into a broker of sort. The backend exposed a generic API to the front end, so that it could request gift card information for a given brand from the backend in a generic way. The frontend didn\u0026rsquo;t care which processor the brand had, and the backend dealt with keeping track of which processor each brand used, and knew how to interact with that processor\u0026rsquo;s API to generate new cards, get balances, etc.\nThis separation between the front end and the backend was great, and it let us develop each one independently without slowing each other down. Over time we eventually integrated with all the major gift card processors, and this made on-boarding a new customer much quicker and easier.\nFirst Customers # When we signed Uno\u0026rsquo;s and Travelocity they didn\u0026rsquo;t want an open loop gift card, they wanted us to integrate with their existing gift card platform. They were the primary cause of the pivot (described above), and to be honest if it wasn\u0026rsquo;t for them, we wouldn\u0026rsquo;t have signed other larger clients. Once we signed them, it wasn\u0026rsquo;t long before we signed O\u0026rsquo;Charley\u0026rsquo;s which lead us to CVS, which then lead us to Home Depot.\nLanding Home Depot was huge, and it took a lot of work from a lot of people. Phil Moran brought it in, and Brian Rigney lead the negotiation and closing of the deal. He worked closely with Madeline Wong from Home Depot to get the contact written and signed. There were times when I didn\u0026rsquo;t think the deal would close, but Brian and Madeline worked really hard, and got it done. Once we landed Home Depot as our anchor tenant, we knew we were on to something.\nFraud # As mentioned above, one of the things we needed to worry about was Fraud. Potential fraud scared a lot of businesses from selling gift cards online. To overcome this issue, we decided to cover all fraud. That means if someone used a stolen credit card to buy a gift card, we would eat the cost of that gift card, and the business wouldn\u0026rsquo;t have any liability. This resinated well with customers, and was a differentiator between us and our competitors.\nThis was all well and good, but unless we had a way to prevent fraud, we would lose a lot of money, and wouldn\u0026rsquo;t be in business very long. When you add fraud prevention tools, you have to be careful. If they are too strict you will be rejecting good orders, and if not strict enough you will be allowing bad orders through.\nTo help us build our fraud prevention tools we hired Pete Kledaras. Pete and I worked together to build the first set of fraud tools. When we compared it to the other tools available from our credit card processor and others on the market, it easily outperformed them. We switched over to using our custom tool, and our fraud rates dropped.\nThe problem with a lot of other tools on the market, is that they were generic, and were too aggressive. They rejected too many good orders. Our tools were built specific to our use case, and because of this we could tune them the way we needed. If we noticed a new way scammers were trying to get around our controls, we were able to easily adjust our tools to stop it quickly.\nAs we got more customers and sold more cards, our tools had more data to analyze and got smarter and thus better. Because we pooled our datasets between our customers it allowed us to make all of our customers safer at the same time. If a scammer attacked customer A, and we put in a fix to stop it, it would automatically prevent that scammer from using the same trick on all of our other customers.\nCrime stoppers # After we launched Uno\u0026rsquo;s, we didn\u0026rsquo;t have much fraud prevention in place yet. Most of the checks were the ones we got from the payment processor, and manual checks on our part. We didn\u0026rsquo;t have a lot of orders, so it was pretty easy for us to check them when they came in.\nOur first case of fraud was with our Uno\u0026rsquo;s account. Someone had bought a gift card with a stolen credit card and then used it almost right away at an Uno\u0026rsquo;s in Florida. A week later, we were sitting at the office on a Friday night when we noticed that someone with the same email address bought another gift card, but this time they used a different credit card. We assumed the credit card was stolen, so Laura Cannon sprung into action. She reached out to the owner of the credit card to confirm if they placed the order. Just as we thought, it was stolen.\nThinking that the person who bought the card would stick with the same MO Modus Operandi (MO) using it at the same restaurant in the very near future, we knew we had to act quick. Laura called the Uno\u0026rsquo;s that the person used last time, and explained what was happening to the manager. The manager then called the police to have them waiting incase the person came in to use the gift card. We all waited at the office hoping that the person would show up. Laura\u0026rsquo;s phone rings, and it is the manager of the restaurant and she tells us that the person was there, and trying to use the gift card. We confirmed that the gift card he was using was the one that was stolen and she passed the information over to the police officer who was there, and he arrested the suspect. When that happened we all cheered so loud, it was awesome, it felt like we were actual detectives for a night. Before we left we cancelled the gift card, and refunded the money back onto the credit card. We also told the person who owned the credit card, to call their credit card company and report it stolen.\nTragedy # CashStar\u0026rsquo;s first CFO was Mitchell Schecter. Since we were pretty small, and didn\u0026rsquo;t have a lot of work for a CFO, he was a part time person, who worked out of his house in Massachusetts, where he was a CFO for a few other companies. Mitchell handled a lot of things for CashStar, everything from hiring, HR, payroll, and paying all of the bills. If it wasn\u0026rsquo;t for him, we wouldn\u0026rsquo;t get paid, and the lights would shut off.\nUnfortunately he died when he had a bicycle accident when vacationing in France. This hit us hard, we were still a small company so to lose someone you worked with closely, really sucked. Because he was the holder of all of the keys for the company, we were both emotionally and financially stuck at the same time. It took us a while to get over his loss, and also get our financial affairs in order. If it wasn\u0026rsquo;t for Mitchell I don\u0026rsquo;t know if CashStar would have done as well as it did, and it is important to remember everything that he did for the company.\nEarly Employee Hire TimeLine: # December 2007 David Stone starts CashStar with Steven Boal Feb 2008 David Wood is hired May 2008 Jeff Dixon hired as a consultant June 2008 Phelps is hired August 2008 Ken Cochrane, Phil Moran, Laura Cannon, and Miklos Van Halen are hired. Brian McLaughlin hired part time as contractor. September 2008 Sheri Hall hired as a contractor. Fall 2008: Pete Kledaras hired January 2009: Brian McLaughlin hired full time. Feb 2009 David Krauth hired April 2009 Brandon Tarbet hired May 2009 Brian Rigney hired as consultant Jun 2009 Richard Franks and Mike Ellis hired August 2009 Patrick Devine and Garry Turner hired Sep 2009 John Costa and Brian Rigney hired full time Oct 2009 Janet Acker hired Fall 2009 Kathleen Goodwin hired Caveat # A majority of the things that I wrote about here, happened over 8 years ago, so I wouldn\u0026rsquo;t be surprised if I\u0026rsquo;m misremembering something, or missed someone important. If I did that, I\u0026rsquo;m sorry, please let me know and I\u0026rsquo;ll do my best to correct it.\nConclusion # As you can see there was a lot of things that we needed to do, and problems to overcome in order to successfully sign our first customers and get through our first couple of years. We wouldn\u0026rsquo;t have been able to do it if it wasn\u0026rsquo;t for the great team that we put together. It just proves that if you have the right people, you can do anything.\nBonus: More photos # Here are some more photos that I found.\nAlex about to erase a whiteboard with Janet in background Bowling outing Dave Wood David Stone press photo Fire down the street Visit to new offices on Middle St Mike\u0026rsquo;s birthday Mother\u0026rsquo;s day contest Team outing mini golf Phelps on Ball Early team dinner Filling Phelps office with Balloons Filling Phelps office with Balloons 2 Filling Phelps office with Balloons 3 Phelps in his office after popping all balloons Phelps using ladder as a chair in a meeting Rainbow as seen from the office Thanksgiving lunch David making first purchase Glenn taking picture of David making first purchase David opening champagne David and Phelps drinking champagne out of plastic cups Our fridge fully stocked Playing Kickball at the castle Sheri breaking her bocce ball My desk Whiteboard during a brainstorm session. Middle St Offices Middle St temp kitchen Middle St temp Offices Middle St Office buildout ","date":"September 17, 2017","externalUrl":null,"permalink":"/2017/09/17/cashstar-the-early-years/","section":"Posts","summary":"","title":"CashStar the early years","type":"post"},{"content":"","date":"September 17, 2017","externalUrl":null,"permalink":"/tags/history/","section":"Tags","summary":"","title":"History","type":"tags"},{"content":"","date":"September 17, 2017","externalUrl":null,"permalink":"/tags/work/","section":"Tags","summary":"","title":"Work","type":"tags"},{"content":"","date":"March 29, 2017","externalUrl":null,"permalink":"/tags/availabilityzones/","section":"Tags","summary":"","title":"AvailabilityZones","type":"tags"},{"content":"","date":"March 29, 2017","externalUrl":null,"permalink":"/tags/aws/","section":"Tags","summary":"","title":"Aws","type":"tags"},{"content":"","date":"March 29, 2017","externalUrl":null,"permalink":"/tags/cloudformation/","section":"Tags","summary":"","title":"CloudFormation","type":"tags"},{"content":"","date":"March 29, 2017","externalUrl":null,"permalink":"/tags/lambda/","section":"Tags","summary":"","title":"Lambda","type":"tags"},{"content":"This is a follow up to my Understanding AWS Regions and Availability Zones in Cloud Formation post that I made a little while ago.\nPreviously I talked about how you can dynamically select the number of AvailabilityZones (AZs) that you want to deploy your instances too, based on the region. Since some regions only offer two AZs and others have more, you will always want to deploy to as many as possible, to make your application better handle an outage if an AZ were to go down.\nOne of the issues that I mentioned was the problem with the Fn::GetAZs function not always returning the correct values. To make things better, I have decided to create a lambda function to make an API call to get the value, and return that. Since the API calls should be more accurate, it will be easier to use, and it will also allow me to standardize the output.\nStep 1: Create lambda function. # We need to create the Lambda function. Here is the python code.\nimport cfnresponse import boto3 def handler(event, context): ec2c = boto3.client(\u0026#39;ec2\u0026#39;) r = ec2c.describe_availability_zones() azs = r.get(\u0026#39;AvailabilityZones\u0026#39;) az_list = [az.get(\u0026#39;ZoneName\u0026#39;) for az in azs if az.get(\u0026#39;State\u0026#39;) == \u0026#39;available\u0026#39;] az0 = az_list[0] az1 = az_list[1] if len(az_list) \u0026gt; 2: az2 = az_list[2] else: az2 = az0 resp = {\u0026#39;AZ0\u0026#39;: az0, \u0026#39;AZ1\u0026#39;: az1, \u0026#39;AZ2\u0026#39;: az2} cfnresponse.send(event, context, cfnresponse.SUCCESS, resp) return resp Nothing too fancy, all that I\u0026rsquo;m doing is importing some libraries, making an API call, parsing the results, and building the response, and sending it back. It is important to note that for my use case, I always wanted 3 results to be returned, if there is only two AZ\u0026rsquo;s in a region, the third AZ (AZ2) will be the same as the first (AZ0). You might not need this, so change it to match your use case.\nStep 2: Add Lambda Function to CloudFormation # All that I am doing is taking the code from above, and adding it into a CloudFormation Resource With some properties.\n\u0026#34;AZInfoFunction\u0026#34;: { \u0026#34;Type\u0026#34;: \u0026#34;AWS::Lambda::Function\u0026#34;, \u0026#34;Condition\u0026#34;: \u0026#34;LambdaSupported\u0026#34;, \u0026#34;Properties\u0026#34;: { \u0026#34;Code\u0026#34;: { \u0026#34;ZipFile\u0026#34; : { \u0026#34;Fn::Join\u0026#34; : [\u0026#34;\\n\u0026#34;, [ \u0026#34;import cfnresponse\u0026#34;, \u0026#34;import boto3\u0026#34;, \u0026#34;def handler(event, context):\u0026#34;, \u0026#34; ec2c = boto3.client(\u0026#39;ec2\u0026#39;)\u0026#34;, \u0026#34; r = ec2c.describe_availability_zones()\u0026#34;, \u0026#34; azs = r.get(\u0026#39;AvailabilityZones\u0026#39;)\u0026#34;, \u0026#34; az_list = [az.get(\u0026#39;ZoneName\u0026#39;) for az in azs if az.get(\u0026#39;State\u0026#39;) == \u0026#39;available\u0026#39;]\u0026#34;, \u0026#34; az0 = az_list[0]\u0026#34;, \u0026#34; az1 = az_list[1]\u0026#34;, \u0026#34; if len(az_list) \u0026gt; 2:\u0026#34;, \u0026#34; az2 = az_list[2]\u0026#34;, \u0026#34; else:\u0026#34;, \u0026#34; az2 = az0\u0026#34;, \u0026#34; resp = {\u0026#39;AZ0\u0026#39;: az0, \u0026#39;AZ1\u0026#39;: az1, \u0026#39;AZ2\u0026#39;: az2}\u0026#34;, \u0026#34; cfnresponse.send(event, context, cfnresponse.SUCCESS, resp)\u0026#34;, \u0026#34; return resp\u0026#34; ]]} }, \u0026#34;Handler\u0026#34;: \u0026#34;index.handler\u0026#34;, \u0026#34;Role\u0026#34;: { \u0026#34;Fn::GetAtt\u0026#34; : [\u0026#34;LambdaExecutionRole\u0026#34;, \u0026#34;Arn\u0026#34;] }, \u0026#34;Runtime\u0026#34;: \u0026#34;python2.7\u0026#34;, \u0026#34;MemorySize\u0026#34;: \u0026#34;128\u0026#34;, \u0026#34;Timeout\u0026#34;: \u0026#34;10\u0026#34; } } In order to run this code you need a few other things which are listed below.\nHere is an IAM role for the lambda Function so that it has the correct API permissions. This allows the ability to write to cloudwatch logs, and make the ec2:DescribeAvailabilityZones API call. Anything else will be blocked.\n\u0026#34;LambdaExecutionRole\u0026#34;: { \u0026#34;Type\u0026#34;: \u0026#34;AWS::IAM::Role\u0026#34;, \u0026#34;Condition\u0026#34;: \u0026#34;LambdaSupported\u0026#34;, \u0026#34;Properties\u0026#34;: { \u0026#34;AssumeRolePolicyDocument\u0026#34;: { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [{ \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Principal\u0026#34;: {\u0026#34;Service\u0026#34;: [\u0026#34;lambda.amazonaws.com\u0026#34;]}, \u0026#34;Action\u0026#34;: [\u0026#34;sts:AssumeRole\u0026#34;] }] }, \u0026#34;Path\u0026#34;: \u0026#34;/\u0026#34;, \u0026#34;Policies\u0026#34;: [{ \u0026#34;PolicyName\u0026#34;: \u0026#34;root\u0026#34;, \u0026#34;PolicyDocument\u0026#34;: { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [{ \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: [\u0026#34;logs:CreateLogGroup\u0026#34;, \u0026#34;logs:CreateLogStream\u0026#34;, \u0026#34;logs:PutLogEvents\u0026#34;], \u0026#34;Resource\u0026#34;: \u0026#34;arn:aws:logs:*:*:*\u0026#34; },{ \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: [\u0026#34;ec2:DescribeAvailabilityZones\u0026#34;], \u0026#34;Resource\u0026#34;: \u0026#34;*\u0026#34; }] } }] } } Now we make a custom resource, which we will use when we want to call our Lambda function from the template.\n\u0026#34;AZInfo\u0026#34;: { \u0026#34;Type\u0026#34;: \u0026#34;Custom::AZInfo\u0026#34;, \u0026#34;Condition\u0026#34;: \u0026#34;LambdaSupported\u0026#34;, \u0026#34;Properties\u0026#34;: { \u0026#34;ServiceToken\u0026#34;: { \u0026#34;Fn::GetAtt\u0026#34; : [\u0026#34;AZInfoFunction\u0026#34;, \u0026#34;Arn\u0026#34;] }, \u0026#34;Region\u0026#34;: { \u0026#34;Ref\u0026#34;: \u0026#34;AWS::Region\u0026#34; } } 3. Make Lambda Conditional # Lambda is a new service, and still isn\u0026rsquo;t available in all of the different regions, because of this, I still fall back to my old way of getting Availability Zones when Lambda isn\u0026rsquo;t supported. To decide if I\u0026rsquo;m going to use lambda or not, I\u0026rsquo;m using a condition statement.\n\u0026#34;LambdaSupported\u0026#34;: { \u0026#34;Fn::Equals\u0026#34;: [ { \u0026#34;Fn::FindInMap\u0026#34;: [ \u0026#34;AWSRegion2AZ\u0026#34;, { \u0026#34;Ref\u0026#34;: \u0026#34;AWS::Region\u0026#34; }, \u0026#34;LambdaSupport\u0026#34; ] }, \u0026#34;yes\u0026#34; ] } This looks a my AWSRegion2AZ Mapping to see if Lambda is available in a given region, and if so, return true.\n\u0026#34;AWSRegion2AZ\u0026#34;: { \u0026#34;ap-northeast-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Tokyo\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;ap-northeast-2\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Seoul\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;ap-south-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Mumbai\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;ap-southeast-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Singapore\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;ap-southeast-2\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Sydney\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;3\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; }, \u0026#34;ca-central-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Central\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;no\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;eu-central-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Frankfurt\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;eu-west-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Ireland\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;3\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; }, \u0026#34;eu-west-2\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;London\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;sa-east-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Sao Paulo\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;no\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;3\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; }, \u0026#34;us-east-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;N. Virgina\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;4\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; }, \u0026#34;us-east-2\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Ohio\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;3\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; }, \u0026#34;us-west-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;N. California\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;us-west-2\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Oregon\u0026#34;, \u0026#34;LambdaSupport\u0026#34;: \u0026#34;yes\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;3\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; } } Step 4. Using the Function # Now that you have everything in place, you can use your custom lambda function. Here is how you use it, inside your template.\nGet the first AZ in this region. # {\u0026#34;Fn::GetAtt\u0026#34;: [\u0026#34;AZInfo\u0026#34;, \u0026#34;AZ0\u0026#34;]} Get the second AZ in this region. # {\u0026#34;Fn::GetAtt\u0026#34;: [\u0026#34;AZInfo\u0026#34;, \u0026#34;AZ1\u0026#34;]} Get the third AZ in this region. # {\u0026#34;Fn::GetAtt\u0026#34;: [\u0026#34;AZInfo\u0026#34;, \u0026#34;AZ2\u0026#34;]} Full example for selecting Availability Zones for a Subnet. # In this example, we will select the Availability Zone for the given subnet.\nFirst, we check to see if LambdaSupported, and if so, we use Lambda. If not, we fall back to our old approach.\n\u0026#34;PubSubnetAz1\u0026#34;: { \u0026#34;DependsOn\u0026#34;: \u0026#34;Vpc\u0026#34;, \u0026#34;Properties\u0026#34;: { \u0026#34;AvailabilityZone\u0026#34;: { \u0026#34;Fn::If\u0026#34;: [ \u0026#34;LambdaSupported\u0026#34;, { \u0026#34;Fn::GetAtt\u0026#34;: [ \u0026#34;AZInfo\u0026#34;, \u0026#34;AZ0\u0026#34; ] }, { \u0026#34;Fn::Select\u0026#34;: [ { \u0026#34;Fn::FindInMap\u0026#34;: [ \u0026#34;AWSRegion2AZ\u0026#34;, { \u0026#34;Ref\u0026#34;: \u0026#34;AWS::Region\u0026#34; }, \u0026#34;AZ0\u0026#34; ] }, { \u0026#34;Fn::GetAZs\u0026#34;: { \u0026#34;Ref\u0026#34;: \u0026#34;AWS::Region\u0026#34; } } ] } ] }, \u0026#34;CidrBlock\u0026#34;: { \u0026#34;Fn::FindInMap\u0026#34;: [ \u0026#34;VpcCidrs\u0026#34;, \u0026#34;pubsubnet1\u0026#34;, \u0026#34;cidr\u0026#34; ] }, \u0026#34;Tags\u0026#34;: [ { \u0026#34;Key\u0026#34;: \u0026#34;Name\u0026#34;, \u0026#34;Value\u0026#34;: { \u0026#34;Fn::Join\u0026#34;: [ \u0026#34;-\u0026#34;, [ { \u0026#34;Ref\u0026#34;: \u0026#34;AWS::StackName\u0026#34; }, \u0026#34;Subnet1\u0026#34; ] ] } } ], \u0026#34;VpcId\u0026#34;: { \u0026#34;Ref\u0026#34;: \u0026#34;Vpc\u0026#34; } }, \u0026#34;Type\u0026#34;: \u0026#34;AWS::EC2::Subnet\u0026#34; }, I hope that was helpful, if you have any questions, hit me up on Twitter @KenCochrane.\n","date":"March 29, 2017","externalUrl":null,"permalink":"/2017/03/29/using-lambda-to-get-azs-in-cloudformation/","section":"Posts","summary":"","title":"Using Lambda to get a users Availability Zone list in Cloud Formation","type":"post"},{"content":"I have been using CloudFormation a lot over the past year with my work on Docker for AWS. It is a pretty good product, but there are still some rough edges that make using it a lot harder then it needs to be. I have compiled a list of changes and feature requests that I would like to see added to CloudFormation to make my life easier. I hope this blog post gets read by the CloudFormation team and helps guide their roadmap. If they have any questions, feel free to reach out to me on Twitter, Linkedin, etc any time.\nIntrinsic Functions # These functions allow you to do limited programming inside of a CloudFormation template. As of this writing, there are currently only 9, not counting Condition Functions. They include Base64, FindInMap, GetAtt, GetAZs, ImportValue, Join, Select, Split, and Sub. They have added a few recently, and I have some suggestions on some more they could add.\nFn::Size # Would accept a String or a List, If a List it will return the size of the list. If a String it will return the length of the string.\nThis could also be broken up into two functions, one for string, and one for list, if that is easier.\nFor List:\n{ \u0026#34;Fn::Size\u0026#34; : [ list of values ] } For String\n{ \u0026#34;Fn::Size\u0026#34; : {\u0026#34;Ref\u0026#34;: \u0026#34;aParameter\u0026#34;} } If you combine this with the Fn::Split function it will allow you to see how many items were entered when someone has a CommaDelimitedList parameter, which is helpful when it comes to Conditionals.\n{ \u0026#34;Fn::Size\u0026#34; : { \u0026#34;Fn::Split\u0026#34; : [ \u0026#34;delimiter\u0026#34;, \u0026#34;source string\u0026#34; ] } } Fn::RandomStr # This would return a random string of given length.\nExample: random string with a length of 10.\n{ \u0026#34;Fn::RandomStr\u0026#34; : {\u0026#34;length\u0026#34;: 10} } This is helpful when you need a unique name for a resource, or you need a temp password. I\u0026rsquo;m aware this wouldn\u0026rsquo;t be a good way to generate a password, but it is better then what is available now, which would be to hard code the password.\nFn::GenPasswd # Building off of the Fn::RandomStr, a way to generate a unique password. The main difference with Fn::RandomStr is the ability to add a salt. Still not great since the salt would be hard coded into the template, but I\u0026rsquo;m sure the security folks can come up with a better approach, that is more secure. I\u0026rsquo;m just throwing it out there, to get the discussion started.\n{ \u0026#34;Fn::GenPasswd\u0026#34; : {\u0026#34;length\u0026#34;: 10, \u0026#34;salt\u0026#34;: \u0026#34;mysalt\u0026#34; } } Fn::Lower # Given a string return the string all in lower case. Great for standardizing input.\n{ \u0026#34;Fn::Lower\u0026#34; : {\u0026#34;Ref\u0026#34;: \u0026#34;aParameter\u0026#34;} } Fn::Upper # Given a string return the string all in upper case. Great for standardizing input.\n{ \u0026#34;Fn::Upper\u0026#34; : {\u0026#34;Ref\u0026#34;: \u0026#34;aParameter\u0026#34;} } Fn::Trim # Given a string return the string with any whitespace before and after removed. Sometimes people add extra spaces to their parameters, and this causes problems, this would allow an easy way to standardize a string, to make sure it doesn\u0026rsquo;t have any extra spaces.\n{ \u0026#34;Fn::Trim\u0026#34; : {\u0026#34;Ref\u0026#34;: \u0026#34;aParameter\u0026#34;} } Fn::Sort # This will take a List, and return it sorted. You can pass in ascending, or descending as parameters along with the list.\n{ \u0026#34;Fn::Sort\u0026#34; : [ \u0026#34;ascending\u0026#34;, [ list of values ] ] } Condition Functions # These are function that are used when building Conditionals.\nFn::In # Will return True if a given string is in the list of values.\n{ \u0026#34;Fn::In\u0026#34; : [ {\u0026#34;Ref\u0026#34;: \u0026#34;aParameter\u0026#34;}, [ list of values ] ] } Here is how it works with a Python example.\n\u0026gt;\u0026gt;\u0026gt; names = [\u0026#39;ken\u0026#39;, \u0026#39;zander\u0026#39;, \u0026#39;maddox\u0026#39;, \u0026#39;emily\u0026#39;] \u0026gt;\u0026gt;\u0026gt; \u0026#34;ken\u0026#34; in names True \u0026gt;\u0026gt;\u0026gt; \u0026#34;charlie\u0026#34; in names False Fn::Empty # Will return True if the given string is empty. It will first call a trim on the string to remove any whitespace.\n{ \u0026#34;Fn::Empty\u0026#34;: {\u0026#34;Ref\u0026#34;: \u0026#34;aParameter\u0026#34;} } The current way around this is to check if the given parameter is equal to \u0026ldquo;\u0026rdquo;, but what if there is \u0026quot; \u0026ldquo;. If we had a trim Function it would make it a little easier, but we don\u0026rsquo;t.\nFix Fn::GetAzs # Right now Fn::GetAzs returns different values based on your account type. If you have an EC2-Classic account it returns all availability zones in the region, even if you don\u0026rsquo;t have access to them. This is inconsistent with EC2-VPC accounts which will only return AZ\u0026rsquo;s they have access too. Because of this, you can\u0026rsquo;t really trust this value, it would be nice if it was fixed.\nParameters # Parameters are one of the main parts of a CloudFormation template, that make it possible to customize the resulting stack, based on the requirements of the user. The current set of parameters are very limiting. I would like to see some changes to improve the UI/UX for the user, as well as some changes to help the CloudFormation developer.\nNew AWS Specific Parameters # There are currently some AWS Specific parameters that are available, that make it easier to fill out the CloudFormation template. This is because it lets you pick values based on your account. These are a great start, but there are a few problems with them. It would also be nice if there were a few more added to make our lives better.\nNew Parameters # Here are a few of the new parameters I would like to see.\nAWS::EC2::InstanceType # Each region allows a different set of EC2 Instance Types, and the list changes all of the time. The current approach is that you have a hard coded list in your CloudFormation template that has this list. You have currently have two options:\nAdd one list for all regions, this might result in a user selecting an instance type in a region that isn\u0026rsquo;t supported. Add a mapping of available instance types by region. Number one is easier to maintain, but is more error prone for your users. Number two is less error prone, but a pain to keep up to date since the list changes all of the time.\nSince AWS knows which instances are available in which region, why don\u0026rsquo;t you provide a new parameter type that dynamically populates with only the available instances in each region. If someone wants a template that doesn\u0026rsquo;t support all instance types, they can still do what they do now, and go with a hard coded list. But in my experience, most people don\u0026rsquo;t care, and they want to offer all available instance types in a given region. Providing this parameter, will make me, and a lot of other people much happier.\nAWS::EC2AccountType # There are a couple of different account types based on when your account was created. You are either EC2-Classic, or EC2-VPC. I wrote a blog post about this a little while ago, to out more, check it out. The main issue is that you don\u0026rsquo;t know what kind of account a user has, and because of this, you don\u0026rsquo;t know if you can trust the results from Fn::GetAzs. If we could find out which account type the user has, we can add some logic into the template to decide the best approach for getting availability zones.\nConditional Parameters # It would be nice if you could add conditions to parameters and only show them if the condition is met. For example, in some regions EFS and Lambda are not available, so if you are in one of those regions, then don\u0026rsquo;t bother showing parameters related to those services.\nConstraints for AWS Specific Parameter Types # The AWS Specific Parameter types are great, but currently you can\u0026rsquo;t add any constraints on those values. Or if you can, I can\u0026rsquo;t find the documentation on how to do it.\nList Parameters # It would be helpful if you could have some constraints for the List parameters.\nFor example, if you offered a List\u0026lt;AWS::EC2::AvailabilityZone::Name\u0026gt; parameter, it would be nice to be able to say pick 2, and if they pick less or more than 2, then throw an error. Right now, they can pick what ever number they want, and you have to deal with that in the template.\nAnother problem is when you offer one of these parameters you need to have a value in it, you can\u0026rsquo;t leave it blank. This causes a problem if you want to offer a template where someone brings their own VPC, and they need to enter the VPC along with the Subnets in each of the AZ\u0026rsquo;s.\nIf you want to support three AZ\u0026rsquo;s in the regions that have three AZ\u0026rsquo;s then you will need to add three AWS::EC2::Subnet::Id fields. But if you do that, then what happens in regions that only have two Availability Zones? You are required to enter a value. It would be nice to allow some of the fields to be empty, to satisfy those cases. Combine that with my other request above where you can have conditional parameters, and that would make it easier for the user.\nAnother issue is that there is no way to cause the parameters to fail validation before the stack is launched. The most common problem is people forgetting to populate AWS::EC2::KeyPair::KeyName. If we could add a validation to these parameters, so that the template doesn\u0026rsquo;t validate unless populated, before the stack is launched, it will help prevent these issues.\nThere is a little hack that works for KeyPair, where you can put a MinLength of 1, but this is undocumented, and may break in the future. It also doesn\u0026rsquo;t work with Troposhere since it is undocumented.\nMin # Pick the minimum number of required selections\nMax # Pick the maximum number of required selections\nAllowEmpty # Allow the parameter to be empty.\nRequired # If specified parameter will be required, and will fail validation.\nExamples # These values can be combined when appropriate.\n\u0026#34;PubSubnetAz1\u0026#34;: { \u0026#34;Description\u0026#34;: \u0026#34;Public Subnet 1\u0026#34;, \u0026#34;Type\u0026#34;: \u0026#34;AWS::EC2::Subnet::Id\u0026#34;, \u0026#34;AllowEmpty\u0026#34;: true }, \u0026#34;AZs\u0026#34;: { \u0026#34;Description\u0026#34;: \u0026#34;Pick your Availability Zones\u0026#34;, \u0026#34;Type\u0026#34;: \u0026#34;List\u0026lt;AWS::EC2::AvailabilityZone::Name\u0026gt;\u0026#34;, \u0026#34;ConstraintDescription\u0026#34;: \u0026#34;must be an Availability Zone\u0026#34;, \u0026#34;Min\u0026#34;: 2, \u0026#34;Max\u0026#34;: 3, \u0026#34;Required\u0026#34;: true } UI/UX changes # Right now there are only two types of input types, text boxes and select boxes. This is ok, but limiting. What happens if you need to enter a license key, or some extra user data shell scripts, it gets really ugly trying to cram that all into a one line text box.\nIt would be nice, if there was a way to specify what type of input you can have on a given parameter. Two of the input types I would really like are text area and checkbox.\nTextArea and Checkbox # You could do this a couple of different ways, you could add a section under the \u0026quot;Metadata\u0026quot;:\u0026quot;AWS::CloudFormation::Interface\u0026quot; section where you can specify the widget type, or you can add to the parameter section directly.\n\u0026#34;License\u0026#34;: { \u0026#34;ConstraintDescription\u0026#34;: \u0026#34;Please enter your license key\u0026#34;, \u0026#34;Description\u0026#34;: \u0026#34;License Key\u0026#34;, \u0026#34;Type\u0026#34;: \u0026#34;String\u0026#34;, \u0026#34;Widget\u0026#34;: \u0026#34;TextArea\u0026#34; }, \u0026#34;AutoUpdate\u0026#34;: { \u0026#34;Description\u0026#34;: \u0026#34;Enable Auto Update?\u0026#34;, \u0026#34;Type\u0026#34;: \u0026#34;String\u0026#34;, \u0026#34;Default\u0026#34;: \u0026#34;no\u0026#34;, \u0026#34;Widget\u0026#34;: \u0026#34;Checkbox\u0026#34; } Widget would only be a valid parameter for Type String, and default to what they have now, if not specified, to maintain backwards compatibility.\nWhen you pick a checkbox widget it will render a checkbox, default will specify if it is checked or not by default. The two values are \u0026ldquo;yes\u0026rdquo; and \u0026ldquo;no\u0026rdquo;. \u0026ldquo;yes\u0026rdquo; for when checked, \u0026ldquo;no\u0026rdquo; when not checked.\nAdvanced section # Some templates get really complicated with lots of parameters, most of the people don\u0026rsquo;t need a lot of these parameters. It would be nice if there was a way to have an advanced section, where you can hide the advanced settings, and have the section collapsed by default. This will make it easier to use for basic users, but still give the advanced settings that pro users will want.\nI know this is possible, since it is currently available on the second page of the wizard today. If we could have the same option on the first page, along with the ability specify which parameter groups go in which section. This could be done with the \u0026quot;Metadata\u0026quot;:\u0026quot;AWS::CloudFormation::Interface\u0026quot;:\u0026quot;ParameterGroups\u0026quot; section. You could add another parameter, that specifies which section it belongs in.\n\u0026#34;Metadata\u0026#34;: { \u0026#34;AWS::CloudFormation::Interface\u0026#34;: { \u0026#34;ParameterGroups\u0026#34;: [ { \u0026#34;Section\u0026#34;: \u0026#34;Top\u0026#34;, \u0026#34;Label\u0026#34;: { \u0026#34;default\u0026#34;: \u0026#34;My first label\u0026#34; }, \u0026#34;Parameters\u0026#34;: [ \u0026#34;firstParam\u0026#34;, \u0026#34;secondParam\u0026#34; ] }, { \u0026#34;Section\u0026#34;: \u0026#34;Advanced\u0026#34;, \u0026#34;Label\u0026#34;: { \u0026#34;default\u0026#34;: \u0026#34;Advanced Properties\u0026#34; }, \u0026#34;Parameters\u0026#34;: [ \u0026#34;advParam1\u0026#34;, \u0026#34;advParam2\u0026#34;, \u0026#34;advParam3\u0026#34;, ] } ] } } Validations # Validate by step # Right now the validations don\u0026rsquo;t happen until the form is completely filled out, it would be nice if the validation happened when they finish from one step to the next, this would go a long way to improve usability.\nBetter Validation # Check the resources that someone has picked, if they are trying to pick something that isn\u0026rsquo;t available in the given region, fail before you start, instead of starting the stack and failing after. For example, if they are trying to use a feature (Lambda, EFS) and it isn\u0026rsquo;t available in the given region, don\u0026rsquo;t even try to start the stack, fail right there. Same goes for instance type, if the instance type isn\u0026rsquo;t available, don\u0026rsquo;t even try to create the stack.\nQuota checks # Before you try to launch the stack, you know exactly what resources are part of the stack, the currently running resources and what the user has for a quota for those resources. You have everything you need to know, if they can start the stack without causing a quota issue. Right now, you allow the stack to get created, and then fail mid stack deploy. You shouldn\u0026rsquo;t allow the stack to even be created if it will fail. It will save the user time, and money.\nVariables # If you want to have some local variables in your template, you can\u0026rsquo;t do it. You could hack the mappings section to store some constants but this isn\u0026rsquo;t the same. It would be nice if there was a way to declare variables in your template, and reference them like you do with parameters and other Ref\u0026rsquo;s. Azure ARM templates support variables, and it is a nice feature I think CloudFormation should add as well.\nThere are two good use cases for using variables. The first is to store constants in the template, like version numbers, etc. The other is a way to store values that are computed, and reference the variable everywhere instead of copying and pasting the calculation that gave you the computed value everywhere.\nHere is an example:\nLets say you have three parameters and based on a condition you need to do something a little different.\n\u0026#34;Fn::If\u0026#34;: [ \u0026#34;HasOnly2AZs\u0026#34;, { \u0026#34;Fn::Join\u0026#34;: [ \u0026#34;,\u0026#34;, [{\u0026#34;Ref\u0026#34;: \u0026#34;PubSubnetAz1\u0026#34;}, {\u0026#34;Ref\u0026#34;: \u0026#34;PubSubnetAz2\u0026#34;}] ] },{\u0026#34;Fn::Join\u0026#34;: [\u0026#34;,\u0026#34;, [{\u0026#34;Ref\u0026#34;: \u0026#34;PubSubnetAz1\u0026#34;}, {\u0026#34;Ref\u0026#34;: \u0026#34;PubSubnetAz2\u0026#34;}, {\u0026#34;Ref\u0026#34;: \u0026#34;PubSubnetAz3\u0026#34;}]] }] Without variables you would need to duplicate this code all over your template. If you had variables, you could store this value in the variable, and then use the variable in it\u0026rsquo;s place. It will reduce the number of potential bugs, and reduces the size of the template, as well as makes it a little faster since the calculation is only done once.\nTo get this done, we could add a new section called \u0026ldquo;Variables\u0026rdquo; and a new built in function called \u0026ldquo;Variable\u0026rdquo;\n{ \u0026#34;Variables\u0026#34;: { \u0026#34;version\u0026#34;: \u0026#34;0.1.1\u0026#34;, \u0026#34;myVariable\u0026#34;: { \u0026#34;Fn::If\u0026#34;: [ \u0026#34;HasOnly2AZs\u0026#34;, { \u0026#34;Fn::Join\u0026#34;: [ \u0026#34;,\u0026#34;, [{\u0026#34;Ref\u0026#34;: \u0026#34;PubSubnetAz1\u0026#34;}, {\u0026#34;Ref\u0026#34;: \u0026#34;PubSubnetAz2\u0026#34;}] ] },{\u0026#34;Fn::Join\u0026#34;: [\u0026#34;,\u0026#34;, [{\u0026#34;Ref\u0026#34;: \u0026#34;PubSubnetAz1\u0026#34;}, {\u0026#34;Ref\u0026#34;: \u0026#34;PubSubnetAz2\u0026#34;}, {\u0026#34;Ref\u0026#34;: \u0026#34;PubSubnetAz3\u0026#34;}]] }] } } } To use it you would so something like this. Similar to \u0026quot;Ref\u0026quot;\n{\u0026#34;Variable\u0026#34;: \u0026#34;myVariable\u0026#34;} Resources # One of my current pet peeves is the inability to get properties for resources that are not part of your stack. For example, if you have a AWS::EC2::VPC::Id parameter in your template, you can\u0026rsquo;t get any properties from that VPC even though you know the Id. It would be nice to be able to get the CIDR, etc and use that inside of the template. If we could do that, it would remove the need of adding yet another parameter to the template.\nConclusion # I hope this blog post finds it\u0026rsquo;s way to the CloudFormation team inside of AWS. If so, I hope they don\u0026rsquo;t consider this blog post as a bad thing, that wasn\u0026rsquo;t my goal. I really like what they have done so far, and I understand how hard it is to maintain a product that is used by a large user base, and keeping everyone happy. They have to deal with keeping the service up and running, as well as implementing internal changes, and then once those are all done, they probably have very little time to add new features.\nIf anyone from the CloudFormation team wants to talk, I would love to chat and answer any questions about the stuff I have mentioned above. You can find me on Twitter, linkedin, and my email address is all over the place on github.\nIf you are also a CloudFormation user and you have ideas for stuff that I haven\u0026rsquo;t already listed, feel free to reach out as well, and I\u0026rsquo;ll update the blog post to include the best ideas.\n","date":"March 25, 2017","externalUrl":null,"permalink":"/2017/03/25/my-cloudformation-wishlist/","section":"Posts","summary":"","title":"My CloudFormation wish list","type":"post"},{"content":"February 15th, 2017 was my 5th year anniversary of joining dotCloud/Docker. I have been thinking a lot about those 5 years at Docker, and thought it would make for a good blog post. This is going to be fairly long, but I think it will be worth it to see how Docker progressed over time.\nHow I got hired # Before I can explain how I got the job, I need to first give a little background. Before working at dotCloud/Docker I worked for a startup called CashStar. At CashStar I was the first engineer hired to build the SaaS product they were developing. When I got there, they only had a powerpoint presentation and a half working demo. This was good because it meant I had the freedom to do what I wanted and pick the technologies that I felt were best for the job. I decided on a typical Linux stack with the following as my base: Linux, Python/Django, MySQL. This stack worked very well for what we were doing. The only change I might make if I had to do it again is switch out MySQL with PostgreSQL.\nSkip forward a couple of years and we had a nice product with big customers, and a good size team to support it. Thanksgiving to Christmas is our busy time of year, and then things slow down after new years. This was a good time to review the previous year and see what our pain points were and start planning our roadmap for the next year. One of the pain points we were having was around deploying our applications to our servers. Around the same time PaaS\u0026rsquo;s started to become popular, and I started to do some research on the different python PaaS\u0026rsquo;s on the market, in hopes of finding out how they work. So that I could remember what I found, I wrote a blog post for each one, with my notes. One of those blog posts was for a new PaaS called dotCloud. I really liked what they were doing, so I started following them, and trying to figure out how they did things so that I could build my own internal PaaS at CashStar.\nThen one day I came across this tweet:\nSo you like automating the deployment and scaling of a large application? How about hundreds of thousands of them? jobs+sre@dotcloud.com\n\u0026mdash; dotCloud (@dot_cloud) December 21, 2011 I wasn\u0026rsquo;t looking for a job, but I\u0026rsquo;m always on the lookout for what is available, because you never know when the next great opportunity is going to present itself. I sent in an email to see if they were interested in a remote employee, and they said they were, and so I interviewed for the job. Over the course of the next week or so, I meet with a few people, and everyone was really awesome, and I was able to get a better understanding on how they built their PaaS. In an interview with Jérôme Petazzoni was the first time I had ever heard of Linux Containers. Not knowing anything about Linux Containers, I quickly did research so that I could be prepared for the next interview in case they asked me anything about them. I learned a few things quickly:\nThere wasn\u0026rsquo;t much documentation, and what documentation there was, was very limited and low level. (not for the faint of heart) This was really cool stuff It was fairly new, and only supported in the latest kernels It was very hard to use Around the same time I was trying to figure out a better way to package python application, so that they were more like Java applications (War files), and the more I read about linux containers, and talking with the dotCloud folks, the more I realized that this is the future.\nAfter talking with everyone at dotCloud, they made me an offer. I had a good thing at CashStar and before I left, I wanted to make sure this was the right move, before I quit my job to work remotely for a small startup in San Francisco, where I only met the people a few times over google hangout. My wife and I were suppose to go to Florida for vacation, we decided to cancel that trip and instead fly to San Francisco on our own dime, to meet the dotCloud folks in person. I wanted to hang out with the team and get to know them a little more, before I decided to take the job. We also wanted to see if San Francisco would be a good place to move too, if it came to that.\nAfter sitting with Solomon Hykes for a few hours while he talked about his vision of the future, I had decided this was the place for me, and I signed my offer letter that night. When I came back home, I gave CashStar my notice.\nDocker Office # The office was pretty sweet, it was setup like a jungle. It was even featured on TechCrunch Cribs. Check out the video It had everything from big bean bags, 100\u0026rsquo;s of plants, and a pet tortoise named Gordon\nGordon the office mascot When I was first hired, everyone was given a Nerf gun, so that you can defend yourself when working in the jungle. This was fun for a little while, but we soon had Nerf darts everywhere, and people got tired of picking them up. We quickly replaced the Nerf guns with headphones for all new employees.\nSite Reliability Engineer (SRE) at dotCloud # My first job at dotCloud was as a Site Reliability Engineer for the PaaS, and Jerome Petazzoni was my manager. There was one other remote person also on the team, his name was Charles Hooper. If you are not sure what an SRE is, or does, there a SRE book that some SRE\u0026rsquo;s from Google just put out, it is a good read, check it out.\nAn SRE at dotCloud was basically responsible for making sure that the platform was always up and running, and if there was an issue, to help fix the issue, and prevent the same issue from happening again.\nHere is what I did on a typical day:\nlogin, check IRC, and email to see what happened the night before. check nagios to see if there are any active alerts. Fix nagios issues check support tickets work on any projects meetings (stand up, etc) I can\u0026rsquo;t remember why any more, but I did something Solomon really liked, and to show his appreciation, he made me my very own SRE badge. This is one of the cooler awards I have gotten at a company.\nProjects # On top of your normal every day tasks, everyone had projects that they worked on to help improve dotCloud. Here are some of the projects I worked on.\npypi mirrors # On dotCloud the way we handled deploys was like this. You would push your code to us, it could be via git, rsync, etc and then we would take that code, read your config file, and build up a container image, which we would use to deploy to the different hosts.\nAs part of your config you can install dependencies, with the python service, this would be done using pip, and it would pull down the python packages from pypi. 5 years ago, there were stability issues with pypi, and they recommended that you used the --use-mirrors flag for pip, this would allow you to use an official mirror so it will cause less damage if one of the mirrors go down. Worse case you try again.\nWe built this into our python service, with retry logic, to handle if the mirror we ended up on failed. This worked well, but for some reason we would end up where some deploys would work, and others would not. After digging into this some more, it was because some of the official pypi mirrors were not in sync with the master. If you picked an older package, you were fine, but if you picked one that was new, it was luck of the draw if you ended up on a mirror that was up to date.\nAfter some research I found out that there was no easy way to see if a mirror was out of date or not. The mirrors were maintained by volunteers and there was no monitoring setup. I quickly took an audit of the mirrors and found out that more than half were out of date, and some hadn\u0026rsquo;t been updated in months. To monitor the mirrors better I created a library, and a website.\nNow that we had visibility into the problem, it was easier to see what was wrong. I contacted the people in charge of the mirrors that were out of sync, and when one of the mirrors had no owner, I took over maintenance on it. After a few weeks, we had all of the mirrors up to date, and with the new website, it would let us know if one of the mirrors got into a bad state again.\nA few years ago, pypi moved away from using mirrors and now uses a CDN, so this isn\u0026rsquo;t as useful, but it was a good learning experience on how all of this infrastructure worked, and it would come in handy when building the Docker registry.\nAppTracker and Packer # At dotCloud, we tried to run as much of the dotCloud infrastructure on top of dotCloud, but eventually there needed to be a layer between the host and the platform. This layer was important, and really hard to upgrade. dotCloud was written as a bunch of different micro services, and when we needed to upgrade the platform, it was a big task.\nOriginally, the way the platform was deployed, was with a fabric script that would bootstrap the node, bring up the services, and add it to the cluster. Once it was up and running, it would start to accept work, and new services would be deployed to it. This fabric script worked fine when there was only a handful of servers, but when you have hundreds of servers it didn\u0026rsquo;t work well.\nHere is how it worked.\nYou would login to special deployment host, we called it the carrier. Make sure you are in a tmux session. pull down the latest code given the tag for the release. run the fabric script. a. This script would run in parallel across all nodes in the cluster. b. It would push the code to the hosts c. It would run the install scripts for each component d. clean up This process would take hours, and if a host failed it\u0026rsquo;s install you wouldn\u0026rsquo;t know until the end, and you would then need to login to the host, investigate, and repush the update after. Since this process took all day, it limited the number of deployments we could do, so we ended up only deploying a couple of times a week.\nMy goal was to make this process much better in the following ways.\nfaster more stable, repeatable, and predictable automatic roll back targeted installs (only install the service on the hosts that need it) more visibility simple to use, easy to automate in future. With those goals, I decided to change the way we deployed the application, and this is what I came up with.\nThere are four main parts:\nAppTracker: the brains that manages the deployments Packer: Builds a self contained install package for the service (tar ball). Grabber: A service that lives on the host, that handles the installation of the service. versions: keeps track of what versions are running on every host. The details for each part are below, but with this new process, I was able to accomplish all of my goals. Deployments went from taking all day, to on average under 2 minutes.\nThe main reasons for this are because of the following:\nInstead of pushing all of the code out to each node, we pull only the code that is needed to each host. downloading is much faster compared to uploading. We build the package once, and store in s3, vs building on each node. If the build fails, we know before we get to the host. We only build and deploy the services we needed vs doing all of them. We added monitoring and notifications to every step of the process so we always know where we were, so if something goes wrong, it is easy to fix. AppTracker # AppTracker is a service that allows you to deploy any of our services to any hosts. You can pick a service version, along with where you want it installed. You can deploy to a whole cluster, or just a subset of nodes, using a regex.\nWhen a request comes in, it will see if it is a recent build, if not will send the request to packer to get built.\nThere was a webhook setup in our version control system, that would automatically trigger new builds for new code. This made it quicker when you want to do a deployment since your code was already built.\nThere was am easy to use website, along with a cli, and web service for automating tasks.\nPacker # Packer had support for python and node.js since those were the two languages we used at the time, but it could easily be expanded to add more.\nA request would come in, it would then pull down the code, and read the config file for that service. It would see what type of build it is (python, node.js, etc). It would then do everything it needed to do, pull down dependencies etc, and build a stand alone package, and tar it up, and store in s3. It would then let app tracker know it was finished, which will then send a message on a pub/sub channel to grabber.\nThis might look familiar to you, it is doing something similar to what docker does today with docker builds for images, and pushing to the registry. Not quite as advanced as Dockerfiles, but along the same lines.\nGrabber # Grabber lives on each host, and it knows which cluster it lives in, along with its hostname. It listens on a pub/sub channel for new requests. When it gets one, it looks to see if the message is for itself, and if so, if the version that needs to be installed is already installed or not. If it is not already installed it will install the new version.\nInstallation is pretty simple, it will pull down the tar file from s3, extract into the given directory. Switch a symlink and then stop the old version and start the new one. It checks to make sure it started up correctly if not, it rolls back to the old version.\nIt sends it\u0026rsquo;s update to apptracker, and the apptracker website in realtime, so the progress of the release. If any of the nodes failed, the person deploying will be able to see the logs from the failure in the website, and easily trigger a retry, or login to the host for a closer look.\nVersions # This is a simple service that looks at all of the hosts and lists which hosts have which version of applications. This data is shown in a table on the appTracker website. With a few clicks it someone can update the versions that are out of sync.\nThere is a daemon on each host that gathers the information about each service and reports it back to the main service every minute. There are also nagios alerts setup so that if some conditions are meet, it will page an on call person.\nServices # The services at dotCloud where written with something called cloudlets A grandfather to the docker image format. It was very hard to use, but at the time, it was the best thing available. We learned a lot of things from this project, and it helped shape Docker to what it is today.\nI can\u0026rsquo;t remember when, but eventually I was responsible for managing the language and database services on dotCloud. This meant that I needed to keep them up to date with new features and versions. This was a real pain for node.js, which seemed to release a new version every other week.\nEventually we started pushing our custom service, which was a generic service that lets you do what ever you want. This became really popular, and got us thinking more about a Container as a service model, verse a PaaS model.\nDiskWatcher and MemoryWatcher # On dotCloud we had some resource limits for people who ran their applications. It was fairly easy to limit memory, but disk was a little harder. We said we had limits, but we rarely enforced them, the only time we did it was when we noticed someone had way more space then they needed and it was going to consume all available space on the server.\nI created a couple of tools that ran on each hosts and reported back to us the disk and memory usage for each service, and then we can group by account. When someone went over the limit, we could reach out to them and let them know that are over the limit and to clean things up. Most of the disk space issues were caused by log files. There were a few applications that would get in a bad state and easily output 100\u0026rsquo;s of GB\u0026rsquo;s of log files in a matter of hours. If we notice this, we normally got paged, went in, clean up the logs, and let the person know there app is acting weird. Since it was the same people every time, we kind of got used to it, and so didn\u0026rsquo;t they. Sometimes we just needed to restart the container, and it fixed itself.\nMinesweeper # For a while we ran a free cluster, for people to try out services. Whenever you offer something for free there is always someone who will try and game it. The two biggest problems we had was running bitcoin miners and minecraft servers. minecraft servers were not against our TOS, but they usually consumed a lot of resources and caused issues for others on the hosts.\nTo fix this problem, we used to login to each host and see what services were using the most resources and if it was a bitcoin miner, we would stop the container, and contact the owner. We would shut it down, they would just start it back up.\nThis got tiring, so I wrote a program that did this automatically. It wasn\u0026rsquo;t long before they had something that checked to make sure it was up, and if it was they would automatically start it back up. We went back and forth like this until, I would hit my limit (5 times), and I would delete the app. If an account has repeat violations, they would be suspended. It was a real cat and mouse game for a while. But because it was automated, it wasn\u0026rsquo;t taking much of our time anymore, and was just an annoyance.\nHackdays # Once a month we would have a hackday, the goal of the hackday was to do something to improve the platform, and work on something that wasn\u0026rsquo;t on your current sprint plan. Here are a few of my hackday projects that I could remember:\nAndrea wearing Hackday hat help.dotcloud.com # A simple web page that had links to all of the different help options for dotCloud.\nstatus.dotcloud.com # When there is a server crash, or an outage, the on call person would get paged, and they would start investigating what needs to get done. They are also responsible for keeping customers in the loop, so they know if they are affected, and when it will be fixed.\nI made a simple website where the on call person could login, pick the servers and services affected, and then update twitter, dashboard.dotcloud.com, status.dotcloud.com, and IRC all at the same time. This was before statuspage.io, and status.io were around. Before this, it would require the person to login to each service one at a time, and post a message. This made it much quicker and easier.\nThis also allowed us to capture metrics to see how many outages we had, and which hosts were the most affected.\nDjangocon 2012 talk about PCI compliant apps # One of my first conference talks was at DjangoCon 2012 in Washington D.C. It was about building PCI compliant web applications. I was also able to talk with people who were using dotCloud, and help answer any of their questions. I also gave out a bunch of t-shirts.\n.@kencochrane explaining the morass of PCI (credit card industry) requirements in his @dot_cloud shirt #djangocon pic.twitter.com/WTJB4k3b\n\u0026mdash; Gabriel Grant (@gabrielmgrant) September 6, 2012 Here is the video if you are interested in checking it out.\nThe Docker Pivot # On March 10th 2013 John Costa, Charles Hooper, and I flew into San Francisco for 2 weeks, to work on a number of dotCloud projects. Little did we know what was in store for us.\nOn Monday morning March 11th, we gathered in the dotCloud kitchen, and Solomon got us together to talk about what we are going to be doing for the next week. We were going to be working on getting Docker ready to launch. Before that day, Docker was an internal side project that Solomon and a few other engineers were working on.\nWe were sponsoring PyCon, and it was the following weekend, so the idea was that we would launch Docker, and then talk with everyone about it at PyCon. It was pretty much all hands on deck. We divided into two teams. One team would keep working on dotCloud, business as usual. The other team would be all hands on deck for getting Docker ready for primetime.\nI was picked for the Docker team. At this point, I hadn\u0026rsquo;t done much with Docker, so I had to come up to speed quickly. We ended up working 15 hour days. We would get to the office around 8-9am and leave 11pm-midnight. We did this every day until we left for PyCon.\nI worked on a number of different things:\nhelping create the docker registry added support for docker login documentation vagrant support testing On any given day I was jumping between python, ruby, go, and Bash. It was a blur, but we got everything ready for PyCon. I was able to reserve Solomon a lightning talk slot, and he was going to use that to introduce Docker with a quick demo. We didn\u0026rsquo;t expect that many people to be at the lightning talk, but the goal was to record it, and then we can show the video. Little did we know the session was packed, and it went well.\nMe (in red) hard at work with Gordon Ken and Sam discussing how to build docker registry Docker launch # Team #dotcloud prepping for lightening talk at #pycon.... Stay tuned for big announcement by @solomonstre pic.twitter.com/7odsaIC9Zc\n\u0026mdash; dotCloud (@dot_cloud) March 16, 2013 Solomon gave the lightning talk, and immediately right after that, a ton of people started coming over asking for access to Docker. At that point we hadn\u0026rsquo;t opened up the github repo, so people started thinking it was vaporware. We started collecting names, and giving people access to the repo on a case by case basis, but that wasn\u0026rsquo;t going to work for long, we needed to launch Docker for real, but that would require a lot more work. Which meant there wasn\u0026rsquo;t much sleep for the next week.\nBloody Monday # On Monday March 18th after a long weekend at PyCon we gathered again in the kitchen at dotCloud. Solomon got everyone together to let everyone know, that PyCon was a success, and everyone loved Docker, and he has made the difficult decision to pivot dotCloud away from the PaaS and towards Docker. There was some layoffs, and unfortunately we lost some co-workers that day, but due to the new direction, we needed to keep things lean, and needed to keep moving forward.\nThe goal was to keep the PaaS up and running, but we would not be adding any more features, we would get it ready for sale, and look for a buyer. We did not want to leave our customers hanging, and since it was making money it made sense to keep it as long as possible. The company was again divided into two, one for dotCloud, the rest on Docker. We were given the rest of the day off to think about the pivot, and to get some rest from a busy previous week. We would come back in on Tuesday and hit the road running, with a goal to have Docker officially launched by the end of the week.\nI was assigned to the new Docker team, but I was still responsible for maintaining the PaaS, so I was working double duty. Luckily the on-call wasn\u0026rsquo;t that bad, and most of the time was dedicated to building out Docker.\n.@KenCochrane \u0026quot;first M in revenue I will take off my hat for a whole day\u0026quot; /cc @docker pic.twitter.com/h7ulip5aOH\n\u0026mdash; Julien Barbier 🙃❤️🏴‍☠️ (@jbarbier) May 23, 2014 Docker, The Future # Once the decision was made to pivot the company, there was a lot to do, in a short amount of time. I was given a few tasks that needed to get done ASAP. One of those was a place where people could upload and share their docker images. We created the docker registry, which was an open source project that anyone could use, but we wanted to have a central location that anyone could use out of the box. We decided to call it the docker index. Since I had a lot of web application experience, I was assigned the task to build it. Since we were a python shop, and I was most familiar with Django the python web framework, I decided to write it in that. We used bootstrap for the UI and I got started.\nDocker Logo # We had a popular open source project, and people liked the name. Now we needed a logo. Thatcher Peskens was tasked with hiring someone to create a logo. He created a contest on 99designs.com and the community was allowed to vote for the winner. Results of the poll\nHere is the blog post announcing the new Docker logo and style.\nThe logo and design was very popular, and we have given away many many thousands of stickers and t-shirts to prove it.\nDocker index # The docker index started off as a fairly basic web application with an API that interacted with the docker registry and the docker engine. It offered the ability to register for an account via the website, or via the docker engine. Once registered, you could push your own public docker images, and share them with the world. Over time, I added features for search, and official images and this made it even easier to get started.\nFor the first year or so, I was the only person who worked full time on the index, every now and then Thatcher would help me with some UX/UI stuff. Eventually we slowly added people to the index team, and even created another team for creating a central service for managing the users of Docker. We called that team docker.io. After a little while I became the engineering manager for both teams and that worked well, and we got a lot done.\nAutomated build system # Solomon had an idea to tie a new Dockerfile feature of docker with the index. He wanted to let people link their github repos to the index, and then when someone made a code change, automatically rebuild that project using the Dockerfile in the git repo, and push to the registry. I liked the idea, and built the first proof of concept during a hack day, and put it in production the next week. It was really popular, too popular in fact, and we had to rewrite it so that it could scale better. Dustin Lacewell took over the project and built version two, and still works on the team that maintains the automated build system today.\nThey have since released version 3 of the system, with a lot more features, and the ability to scale even more.\nBuild Triggers # The automated build system was pretty popular, and people started using it in their CI/CD chain, and because of this, they wanted more flexibility for when the automated build would be triggered. Some people wanted to run their unit tests first, and then only build the image if they passed. With the current setup, that wasn\u0026rsquo;t possible, so I created Build Triggers, which allows you to hit a special endpoint with a HTTP POST and it will trigger your automated build directly.\nRepository Links # One of the things that we recommend is to base your custom images on our official images, this way you build on top of a known good supported image. The problem with this was people were not updating their images when the official images were updated. This mattered when there were security patches, added to the base image. I created a way for people to link their repositories to another one, and if that one changed, it would automatically trigger a rebuild of the child image.\nThis was also another very popular feature, and with it caused the Automated build team some headaches, because whenever an official images was updated, it would trigger a ripple effect and stampeded the automated build system with a ton of builds all at once. They have since improved the scalability of the automated build system to better handle these events. We also put in place some communications between the official image team and the automated build team, so that they give them a heads up when a release is coming, so they can make sure they have enough capacity to handle the load.\nCollaborators # This allowed you to give access to another user to push to your image, and was pretty popular, but people wanted more. So we started working on the organizations and teams feature.\nDocker Hub # Right before the first DockerCon in San Francisco we decided to rebrand the docker index into the Docker Hub. This required a lot of work on merging two different websites (docker.io and index.docker.io) into one website hub.docker.com, and making sure the UI/UX was the same between the two sites. This took a lot of work, but we were able to make the move, and it made the website look much nicer and added some features that we had been working on for a little while.\nPrivate repositories # From the beginning of docker index, I had the concept of a private repo, this allowed you to push up an image, and not make it public. I had only exposed it internally to docker employees to try out, and to make sure it really worked as advertised, we wanted to make sure only the people you wanted to give access too, could access it. This was also a feature we wanted to charge for, so in order to get this working, we had to add a billing component.\nBilling # Before we could start charging for the hub we needed a way to collect money, so I built a billing service that allowed people to subscribe to a plan and they would be automatically renewed at the end of their billing cycle. After going back and forth with Recurly and Stripe, we ended up with Recurly because of a few features strip didn\u0026rsquo;t have at the time. Once Billing was complete, we were able to start charging people to use the hub, and work our way to start paying for the huge AWS bills we have been generating due to all the traffic for the public repositories.\nOrganizations and Teams # Once we released private repositories we needed a better way to share them with others, so we also released a way to create an organization account and to group people into teams and share things based on the team instead of someone individually. We also added a way to convert an user account into an organization account. Most of this work was done by (Josh Hawn, Thatcher, and John C) as part of the docker.io project, and I helped integrate it with the docker index code.\nDocker Hub 2.0 # When we started working on the docker hub, our goal was to start breaking up the monolithic application into smaller micro services, and we started doing that, but one of the issues that was causing us problems was the front end. Serving a common front end across multiple services wasn\u0026rsquo;t ideal. The way you get around this, is that you break up the front end from the backend services and use APIs between the two. Hub 1.0 had a deadline that couldn\u0026rsquo;t be moved, so we were not able to do it the ideal way. So the goal was to get Hub 1.0 out the door, and then work on Hub 2.0 which was breaking up the hub frontend and backend, and release the new frontend.\nOne issue was that most of the people working at Docker were backend folks, we had very little front end experience. So we went out and hired some people who knew what they were doing. We first hired Chris Biscardi , then hired Nathan Hsieh, and Arunan Rabindran. The three of them along with some designers, Nick Kraly and Josh South designed and built the front end, while we worked on providing the backed APIs to them.\nIt took a lot longer then we had planned, but we finally launched a hub 2.0 beta, and then switched over to the new hub 2.0, which is the hub that you see today. For more details, there is a great blog post from Chris UI at Docker.\nDocker Hub Scale # One of the issues that we had to constantly deal with was docker was growing way faster then anyone expected, and with that the demand on the docker hub. The amount of docker pulls was growing and along with it the bandwidth. Here is a little chart showing how it has grown over the years.\nDocker Hub growth curve (June 2015) Docker Cloud # In October 2015 Docker acquired Tutum and I was part of the group of people who did the due diligence on the company, and in doing so, I became very familiar with their products. I was also the most familiar with the Docker Hub, so it just made sense if I was the one that helped integrate the two products into what we now call Docker Cloud.\nThe Tutum folks were based in Spain and New York, and since I was in Maine, that the same timezone as New York, and was only a 5 hour difference compared to the 8 hours in San Francisco. I worked with them to get them access to everything they needed so that we could have a nice tight integration between our two services.\nOne of the things that we needed was a way to charge people (metered billing) for this new service, and current billing system wasn\u0026rsquo;t setup for metered billing. Since I wrote the first version of the billing service it only made sense that I work on the second version.\nI created a new micro service called billing_v2 that had all the original billing features that the hub still used, and added the new features that docker cloud needed, and then we switched the hub and the cloud to use the new billing service. This was a big task in a short amount of time, so I was also able to get some help from Matt Tescher, and John Costa.\nOnce the billing feature was complete and the cloud/hub integration was done, we went live with Docker Cloud.\nConferences # Since I was one of the only employees on the east coast I was able to go to a lot of cool conferences and talk about Docker. Here are a few of the talks I have done over the years.\nDjangoCon US 2013 # I was able to give an intro to Docker talk at DjangoCon US 2013 in Chicago. I actually met Dustin Lacewell after my talk, and convinced him to apply for a job at Docker. Here are my slides, and blog post and video:\nRackspace Solve # Our CEO Ben Golub gave a talk at the Rackspace solve conference in San Francisco, and it was very popular, and they wanted to have the talk at their New York conference. Ben wasn\u0026rsquo;t able to attend, so I was able to fill in for him.\nThis was my first conference talk that wasn\u0026rsquo;t in front of a room full of developers. This was targeted more for C level executives, so I had to dress up (I couldn\u0026rsquo;t wear my hat!), and I couldn\u0026rsquo;t hide behind a podium.\nSince the talk with in NYC, I decided to fly down in the morning, and then fly back at night. RackSpace sent a car to pick me up at the airport. This was also a first, I normally take the subway into town, but I didn\u0026rsquo;t want to turn down a ride right to the door, and it would give me time to practice my talk one last time. The problem was that there was so much traffic, it took me 2 hours to get to the conference, and I arrived about 5 minutes before I was scheduled to go up on stage. Just enough time to go to the bathroom, get mic\u0026rsquo;d up and jump on stage.\nBecause I was in such a rush to get up on stage, I wasn\u0026rsquo;t aware that they had me scheduled for a 30 minute talk, when originally we agreed I was going to have a 40 minute talk. This caused a problem, I had prepared a 40 minute talk, and needed to give it in a 30 minute slot. Nothing more complicated then trying to dynamically remove 10 minutes from your talk on the fly. I think I did alright, but it did prevent me from giving my best. Here is more details about my talk, and the video.\nPyCon 2014 in Montreal # I gave a poster session on Docker at Pycon in Montreal. This was pretty cool, it had been a while since I had been to Montreal, so it was nice to see the city, and help folks with docker. I ran into Nick Lang again, and convinced him to attend DockerCon, and eventually convinced him to come and work with me at Docker.\nI came to the conference with about 50 t-shirts, that was all I was able to carry in my luggage. When I got there, I put the box down, and turned around to talk with the large crowd that gathered, and before I knew it, I turned around and all of the shirts were gone. It was crazy how fast they went.\n@KenCochrane draws a crowd @docker poster session #pycon #pycon2014 pic.twitter.com/fekUwt6uWA\n\u0026mdash; Brad Clements (@bkc) April 13, 2014 DockerCon 2014 # DockerCon 2014 was our first DockerCon, and it was based in SanFrancisco, and it was a lot more popular then we expected. It was a lot of fun. I helped with the conference party in an art gallery, and we had a photographer taking everyones pictures. The picture of John and I jumping for joy is shown below.\nI didn\u0026rsquo;t have to give a talk at this conference, but I helped to create the live demos, and also launch the docker hub.\ncheck out @johncosta and I jumping for joy at the #dockercon after party. pic.twitter.com/sMcWCYmtZB\n\u0026mdash; Ken Cochrane (@KenCochrane) June 12, 2014 Even the @docker investors are having a blast at #dockercon @dschol @trinityventures @DockerCon pic.twitter.com/wrVSIetqY5\n\u0026mdash; Ken Cochrane (@KenCochrane) June 10, 2014 DockerCon Europe 2014 # In 2014 we had DockerCon Europe in Amsterdam, and I had the change to attend. It was my first trip to Europe, so I wasn\u0026rsquo;t sure what to expect. It was great, and I had a lot of fun. I hope to bring the family back sometime in the future.\nOn the way to DockerCon CoreOS released a blog post that caused a stir in the Docker community. Some people called it the start of the container wars. This wasn\u0026rsquo;t the way most people expected to spend the first few days before the conference, but as a company, we all came together and still put on a good conference. We also learned a lot that day, about how to, and how not to handle tough situations as a company.\nHere is a video of the talk that I gave about the Hub. One thing I really liked, was all of the different types of food I normally can\u0026rsquo;t get back home. Probably the best reason to travel, if you ask me :)\nhmmm Back Bacon in Amsterdam DockerCon US 2015 # DockerCon US 2015 was back in San Francisco, and once again, I was able to attend. We launched the Hub 2.0 beta at this dockerCon, and we spent a lot of time getting it ready, and it was well received. I also gave a talk with bc Wong here is the video. DockerCon Europe 2015 # In 2015 DockerCon Europe was in Barcelona Spain, and lucky enough Tutum\u0026rsquo;s offices were not too far in Madrid. So the plan was to go to DockerCon and then take the train over to Madrid to work with the Tutum folks out of their office for a couple of days and then fly home.\nThis was the first conference in a while I didn\u0026rsquo;t have to give a talk, but I made up for it, by helping with other tasks. One of the tasks was stuffing all of the different goodie bags that we hand out to each attendee. I also helped with registration, staffed the docker booths, and went to one of the advisory board meetings.\nAs a gift for every person who attended DockerCon we had a partnership with Yubico to give everyone a yubikey.\nThe problem was that the Yubikeys that we were going to give to everyone were stuck in customs. To solve this, they had someone fly to Spain with a new batch in their carry on luggage, and hand deliver to the conference. The problem was the new ones were not assembled so we got some volunteers and we put them together.\nBuilding yubikey packages Once that was done, we needed to tap them to the chairs in the auditorium where the guests wouldn\u0026rsquo;t be able to see them until we said, reach below your seat. This was our Oprah moment.\nThere was just an Oprah moment at #dockercon @docker gave everyone a new @Yubico pic.twitter.com/Huolqi1Hkg\n\u0026mdash; Ken Cochrane (@KenCochrane) November 16, 2015 DockerCon Seattle # I wasn\u0026rsquo;t able to attend DockerCon US 2016 in Seattle, but the project I was working on, was launched there. Check out the keynote video, to see Docker for AWS.\nMeetups # Besides conferences I also gave a bunch of talks at different meetups around the north east, and I even started a few meetup groups myself. Docker New York City, Docker Boston, and Docker Maine.\nWhile I was in the NYC for the first Docker NYC meetup, I stopped by the offices of Digital Ocean, and helped them build a Docker one-click-app to make it easy to use Docker on their service. It was really nice of them to let me hangout in their office for the afternoon. To thank them, I sent them all Docker shirts.\nKen with Ben and Moisey Uretsky Digital Ocean co-founders Digital Ocean team wearing Docker shirts Docker Core # For the past 3 years I had been working on the Docker registry, index, and hub. When I was working on the cloud, and getting my hands dirty again, I realized I really missed that. I liked being an engineering manager, but I liked being an engineer even more. I also realized that I needed to start working on something different, and so after talking with Solomon and Sam Alba, I was able to move over to the Core team and help them where they needed help.\nDocker 1.11 release # One of my first tasks on the Docker Core team was to help with the Docker 1.11 release. Docker 1.11 was the first release that was built on top of containerD, and because of this, the docker packaging needed to change to support installing multiple binaries. It had been a while since I had worked on rpm and Debian packaging, but I dove in head first and did what I needed to do to help get the release out the door. This was a lot of fun, and got be reintroduced to the docker engine.\nDocker CS releases # Docker CS releases are releases that we commercially support, and back port patches so that companies don\u0026rsquo;t have to upgrade as fast as they might want too. I helped with the CS releases by documenting and updating some of the tools that they use. I didn\u0026rsquo;t work on this project for that long, but it gave me insight into how this all worked, with was very help information.\nDocker for AWS # For the past year I have been working on the editions team building out the Docker for AWS product. This has been a lot of fun, I have learned a lot about AWS, more then I ever thought was possible. I was also able to go to AWS Reinvent in Las Vegas for the first time. This was the largest conference I have ever been too, it was spread out across 3 different casinos on the strip, and it took 25 minutes to walk from one set of talks to another. One day maybe DockerCon will be that big?\nWhat\u0026rsquo;s Next? # Now that I have been with Docker for 5 years, I sometimes wonder what will I work on next. I\u0026rsquo;m not really sure, but I\u0026rsquo;m sure it will be awesome. Sorry for the long post, but I guess it shows all that we have done in the last 5 years, and I\u0026rsquo;m sure I missed a lot as well.\nIf you liked this post, you would probably like Jérôme Petazzoni\u0026rsquo;s Post from dotcloud to docker.\n","date":"March 24, 2017","externalUrl":null,"permalink":"/2017/03/24/5-years-at-docker/","section":"Posts","summary":"","title":"5 Years at Docker","type":"post"},{"content":"","date":"March 24, 2017","externalUrl":null,"permalink":"/tags/dotcloud/","section":"Tags","summary":"","title":"Dotcloud","type":"tags"},{"content":"","date":"December 30, 2016","externalUrl":null,"permalink":"/tags/clynk/","section":"Tags","summary":"","title":"Clynk","type":"tags"},{"content":"","date":"December 30, 2016","externalUrl":null,"permalink":"/tags/companies/","section":"Tags","summary":"","title":"Companies","type":"tags"},{"content":"","date":"December 30, 2016","externalUrl":null,"permalink":"/tags/garbage-to-garden/","section":"Tags","summary":"","title":"Garbage-to-Garden","type":"tags"},{"content":"","date":"December 30, 2016","externalUrl":null,"permalink":"/tags/maine/","section":"Tags","summary":"","title":"Maine","type":"tags"},{"content":"I live in Maine, it\u0026rsquo;s a great state for a lot of things, but [running a business] (http://www.pressherald.com/2016/11/16/forbes-maine-second-worst-state-for-business/) isn\u0026rsquo;t one of them. There are a number of reasons, but the major ones are usually high taxes, and a lack of skilled workers. Because of this, there aren\u0026rsquo;t that many new companies that come along, so I\u0026rsquo;m always on the lookout for companies that are doing cool things. I like to keep an eye on their progress and try and help them out however I can, so that they can succeed and hopefully inspire other people to start more businesses in Maine.\nRecently I find myself talking about this to more and more people, and figured it would be a good blog post. I\u0026rsquo;m hoping that this blog post will introduce me to more great companies that I haven\u0026rsquo;t heard about yet. So if you know of a company I haven\u0026rsquo;t listed here, please let me know. (@KenCochrane on twitter)\nHow I find interesting companies in Maine # There are a few ways I find out about new companies, here are the most popular:\nGreenlight Maine Portland Press Herald, Bangor Daily News Maine Biz Friends and Family Local Meetups Twitter 5 Maine Companies to watch in 2017 # There are a lot of cool companies, but I decided to narrow it down to the top five for this post. In the future I might expand the list, but for now, lets just focus on these five, in no particular order.\nRedzone wireless # Website: RedZoneWireless.com Twitter: @redzonewireless About: Redzone wireless is a new 4G wireless ISP based in Rockland Maine. In Maine there isn\u0026rsquo;t that many options for broadband internet. You have a cable option (if you are lucky), a slower DSL option, satellite, or a slow dial up connection. Due to Maine\u0026rsquo;s large geographic area and small population, the broadband companies are not going to be deploying fiber gigabit connections to the home, anytime soon (if ever). This makes sense for business reasons, because the cost of running fiber to a house will never pay for it self over the lifetime of that connection. That leaves you with a few options. Stick with the existing wires that are already coming to your house (cable, phone, power), or use something that is wireless (cell phone or satellite).\nThe advantage of wireless connections is that you don\u0026rsquo;t need to run the wires to the house, so you can acquire customers a lot cheaper then wired companies. There is still overhead for these companies, you need to build your network (build towers, launch satellites, etc), but once you have your network in place, you can sign up customers quicker, since there is no need to run the cable to their house.\nThe current satellite options are not very good, there is usually a high lag between communication, which is ok for email, web browsing, but doesn\u0026rsquo;t work for video conferencing, or games. There is also usually a bandwidth limit, with crazy overage charges. The costs are also usually much higher because the satellite companies need to re-coop the cost of putting satellites in space, which isn\u0026rsquo;t a cheap undertaking. There are some promising new companies getting into the act, SpaceX being one of them, but they are still a few years away.\nIn a large part of Maine people have faster internet connections on their phones then on their home computers. You can tether off your phone to your computer, but the cell phone companies charge a lot for their data plans, and they usually have bandwidth limits, with large overage fees. If you had a company that could provide a wireless network using the same technology as the cell phone companies, but provided a reasonably priced plan with no bandwidth limits, you could reach a lot of people in Maine, for a lot cheaper then the other options.\nThis is where RedZone Wireless comes in, they are doing just that. They are building out a network a little at a time, and adding the customers along the way. They have recently signed a deal with the company that runs the Maine fiber network, so they can have a fast connection to most of Maine.\nI really like what they are doing, and I think this is the future for getting more Mainers broadband connections in rural areas, and charge a reasonable rate. As the wireless technology improves they will be able to upgrade their towers and increase speeds over time. So the network, just keeps on getting faster. I will be keeping an eye on them, and wishing them luck. They are now available in my area, but unfortunately their speeds can\u0026rsquo;t match my current plan with Time Warner Cable (TWC). But, I\u0026rsquo;m hoping with time, they will be able to beat TWC, and I can switch over.\nIf there is anyone from RedZone wireless reading this, please let me know if there is anything I can do to help you out.\nGarbage to Garden # Website: https://garbagetogarden.org/ twitter: @garbagetogarden About: Garbage to Garden offers a curb side compost pickup in the Portland area. When we moved a few years ago to a new town, we moved to a town where you have to pay by the bag to throw away your garage. This is a good way to force people to reduce the amount of garbage they throw out, by increasing the amount they recycle and compost. We had already been recycling for years, but due to lack of space at our last house we never got around to composting. Our new house has lots of room to compost, so I started researching what I needed to get started, and realized it was more work then I originally thought. While researching composting, I came across Garbage to Garden. They give you a white bucket that you put your food scraps in during the week, and then you put the bucket on the curb with your garbage once a week. They come by on trash day, and they take your old bucket and leave a new clean one. You also have the option of getting 1 bag of compost dropped off at the time of pickup. This gets you the compost that you would have gotten from your own compost pile, with a lot less work.\nI have been using their service for over a year now, and I can\u0026rsquo;t be happier. I get the benefit of paying less for my garbage, get a weekly delivery of compost, and I\u0026rsquo;m helping the environment at the same time. They are slowly expanding their service, if they don\u0026rsquo;t have coverage in your area, let them know you are interested, and they can let you know when they are available in your area. I\u0026rsquo;m hoping that one day, every town in Maine will offer composting as 3rd waste option (along with trash and recycling), and have it managed by Garbage to Garden.\nI really like what they are doing, and I wish them the best of luck going forward. They have won a number of business contests, including Greenlight Maine, so it looks like, I\u0026rsquo;m not the only one that likes what they are doing.\nClynk # Website: https://www.clynk.com/ Twitter: @thinkclynk About: Clynk provides an easy way to recycle your bottles and cans. Clynk provides a service that makes it easy for you to return your bottles and cans and collect your deposits back. They have a deal with Hannaford supermarkets where they replace all of their bottle return departments. Instead of having to deal with counting your bottles, or putting them in a machine that counts and crushes them. Instead you buy some plastic bags that you bring home, fill up with your returnables, stick a Barcode sticker on the bag, and bring back to the store and scan your bar code. They collect the bags of bottles and bring them to their warehouse where they count, sort, and crush them via big machines, that can process bottles and cans much faster then any human. After they count the cans, they will deposit your money in your Clynk account which you can then withdraw from, and get your money. The whole process is so much cleaner and easier then before.\nThey have recently expanded outside of Maine and into New York, so it looks like things are going well. It looks like they have an exclusive deal with Hannaford, which is good to get going, but I wonder if that will limit their growth in the future once they have deployed to all of Hannford. Since Hannaford is pretty big, and part of a larger company, I\u0026rsquo;m sure it will be a while before that happens.\nI would love to see their operation in action, I wonder if they have tours? I wish them luck, going forward, and I hope 2017 is a great year for them.\nRanger Solar # Website: http://www.rangersolar.com/ Twitter: @rngrsolar About: A Maine Solar energy company I am a big fan of solar power, and I would love to add solar panels to my house and start becoming more energy independent. I was already to do it last year, but then all of a sudden the rules for home solar were about to be revoked. The Maine PUC was looking to change the current net metering rules, and one of their proposals where not good for home owners. It would make it much more difficult to make your money back on your solar panels. Since this is still in flux, I decided to wait it out, until this gets more defined. I\u0026rsquo;m sure I\u0026rsquo;m not the only person doing this. The PUC wants to discourage the small home solar installations and encourage larger installations which are much more efficient and provide power cheaper.\nThere is another way to get the benefit of solar power, but also have less risk. You could join a solar garden, sometimes called a Community solar farm. Instead of putting the solar panels on your roof, you and a bunch of other people get together buy a bunch of solar panels, and put them together in another location. Since you have a large amount of solar panels, and you are deciding on a location, you can make sure you pick the right location, and you can get a better deal when selling your power. You can have the amount deducted from your power bill the same way as when the panels are on your house.\nMaine has a few companies (Revision Energy, Assured Solar Energy, etc) that do solar installs for homes and businesses, and that isn\u0026rsquo;t what Ranger Solar does, they are more focused on the larger solar installations. They have one installation at the sanford airport, and they are working at other large scale installations.\nI find this a really interesting market, and one that is getting bigger. We are seeing more and more towns looking at installing solar (Rockland, Portland, south portland, etc) on their property to offset their energy costs. I wish the PUC would allow both larger installations, and smaller home installations. It would spread the grid out a lot more, and if there is a major storm and wires go down, it would help limit the outages since each house has its own power source.\nSince Ranger Solar is a new company, I\u0026rsquo;ll be keeping an eye on them going forward to see what they do next.\nSurge Hydro # Website: http://www.surgehydro.com/ About: Surge Hydro is a hydro electric company I have always loved dams, I grew up near a few dams that I drove by almost daily when I was a kid, and I loved the idea that you can use the river to create power. I even dreamed about owning a small one, one day, but didn\u0026rsquo;t think it was possible since all of current ones were probably owned by large corporations, and you don\u0026rsquo;t hear about people putting in new dams. In fact it is the opposite, you hear a lot about how dams are getting taken down to improve fish migrations. That is why I was surprised to hear about Surge Hydro a little while back. The company was founded by a couple of young guys who saved up their money and bought a dam. How cool is that! Best of all, unlike solar power, hydro power runs night and day so it is a more reliable source of power, assuming the rivers don\u0026rsquo;t dry up.\nIt is nice to see this sort of thing, it is great for the environment, and is reusing something that might have just been sitting there rotting. On their website they say that there are 80,000 dams in the US that don\u0026rsquo;t produce power, if we could convert just a fraction of those, we could create a lot more clean power, and wouldn\u0026rsquo;t need as many power plants. I hope Surge Hydro can help convert these dams so they can generate power, and I wish them the best of luck.\nConclusion # I wish all of these companies the best of luck in 2017, and if there is anything I can do to help any of them, please let me know. I have only listed 5 companies, but there are many more I haven\u0026rsquo;t listed. If you know about another company, please let me know, and I\u0026rsquo;ll try to list the best of them in a future blog post.\n","date":"December 30, 2016","externalUrl":null,"permalink":"/2016/12/30/maine-companies-to-watch-in-2017/","section":"Posts","summary":"","title":"Maine Companies to watch in 2017","type":"post"},{"content":"","date":"December 30, 2016","externalUrl":null,"permalink":"/tags/ranger-solar/","section":"Tags","summary":"","title":"Ranger-Solar","type":"tags"},{"content":"","date":"December 30, 2016","externalUrl":null,"permalink":"/tags/redzone-wireless/","section":"Tags","summary":"","title":"Redzone-Wireless","type":"tags"},{"content":"","date":"December 30, 2016","externalUrl":null,"permalink":"/tags/surge-hydro/","section":"Tags","summary":"","title":"Surge-Hydro","type":"tags"},{"content":"I recently talked about my work on Docker for AWS while working on this project I had to do a lot of research around AWS Regions and Availability Zones, and how they work with CloudFormation. Here are the notes that I gathered.\nWhat is the difference between an AWS Region and an Availability Zone? # The easiest way to answer this questions is to start backwards.\nServers live in a data center. One or more data centers make up an Availability Zone. Two or more Availability Zones in a given area, make up an AWS Region. If a region was a Hotel, the data center would be the hotel room, and the hotel floor would be the Availability Zone. All of the hotel floors are in the same Hotel.\nAvailability Zones are usually close enough to each other so that their is limited delay between network calls between the two AZ\u0026rsquo;s. But they are far enough away where an issue with one (power outage, flood, fire, etc) would not affect the other. They usually have separate utilities (power, internet, etc), and don\u0026rsquo;t depend on either other. This way if one goes away, the other can keep going, like nothing happened.\nHere is a link to the current list of AWS regions and the number of availability Zones in each region.\nThe availability zone lie # If you look at the link above, you will notice that it will show the number of availability zones for each region, but that is kind of a lie. Just because a region has that many availability zones, doesn\u0026rsquo;t mean you have access to them. For example, us-west-1 Northern California says it has 3 availability Zones, but if you log into your EC2 dashboard and look at which AZ\u0026rsquo;s you have access too, you most likely will only see 2. The same goes with us-east-1 , you will see it say 5, but you will only have access to 4.\nIt can do this for a number of reasons, maybe one of the AZ\u0026rsquo;s is running out of space, and they stopped allowing new accounts use that Zone. Or maybe in the case with us-east-1, you really don\u0026rsquo;t need 5 AZ\u0026rsquo;s in one region, and so they spread out the accounts across the five, randomly assigning new accounts to 4 of them.\nEither way, you should double check to make sure you know how many you have access too, and not rely on marketing materials.\nmy us-east-1a is not equal to your us-east-1a # Another trick that AWS does is they randomly assign the AZ names between accounts. So the AZ that points to us-east-1a in my accounts, is probably different then the one in your account. I think it does this so that in case everyone puts everything in us-east-1a it will at least spread the load between the different AZ\u0026rsquo;s in the region. But I\u0026rsquo;m sure there might be other reasons as well. Keep this mind if you are trying to setup services between accounts, you might think you are loading servers in the same AZ, but they might not be. Since the AZ\u0026rsquo;s are next to each other, and the network between them is super fast, you shouldn\u0026rsquo;t have to worry.\nEC2-Classic vs EC2-VPC AWS Accounts # If you have an AWS account that was created before December 4th, 2013 you have what is known as an EC2-Classic account on regions where you have previously deployed resources. EC2-Classic accounts don’t have default VPC’s or the associated subnets, etc. This causes a problem when using a CloudFormation template that uses the Fn::GetAZs function they provide to determine which availability zones you have access too. When used in a region where you have EC2-Classic, this function will return all availability zones for the region, even ones you don’t have access too. When you have an EC2-VPC account, it will return only the availability zones you have access to.\nThis makes it hard to develop a CloudFormation template where you want to automatically assign AZ\u0026rsquo;s, since it will have different functionality for different users.\nA common CloudFormation error looks something like this:\n“Value (us-east-1a) for parameter availabilityZone is invalid. Subnets can currently only be created in the following availability zones: us-east-1d, us-east-1c, us-east-1b, us-east-1e.”\nThe error above would happen if you have an EC2-Classic account, and you don’t have access to the a and b availability zones for that region.\nThere isn’t anything we can do right now to fix this issue, we have contacted Amazon, and we are hoping they will be able to fix the function, or provide us with a way to determine if an account is either EC2-Classic or EC2-VPC, so we can act accordingly.\nHow to tell if you have this issue? This AWS documentation page will describe how you can tell if you have EC2-Classic, EC2-VPC or both. http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-supported-platforms.html\nHow to fix: There are a few work arounds that you can have people try, but they aren\u0026rsquo;t great.\nUse a region that doesn’t have EC2-Classic. The most common region with this issue is us-east-1. So try another region, us-west-1, us-west-2, or the new us-east-2. These regions will more then likely be setup with EC2-VPC and you will not longer have this issue.\nCreate an new AWS account, all new accounts will be setup using EC2-VPC and will not have this problem. You can try and contact AWS support to convert your EC2-Classic account to a EC2-VPC account. For more information checkout the following answer for “Q. I really want a default VPC for my existing EC2 account. Is that possible?” on https://aws.amazon.com/vpc/faqs/#Default_VPCs\nHelpful links:\nhttp://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/default-vpc.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-supported-platforms.html http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-vpc.html https://aws.amazon.com/vpc/faqs/#Default_VPCs https://aws.amazon.com/blogs/aws/amazon-ec2-update-virtual-private-clouds-for-everyone/ Suggested Improvements # Change Fn:GetAZs to only return Availability Zones you have access too, regardless of which type of account you have. Add a CFN function that would let you know if an account in a region was an EC2-Classic or EC2-VPC, so that you could code around the issue yourself. How to configure a CloudFormation template to work on all Regions regardless if they have 2 and 3 Availability Zones. # Due to the way Docker Swarm works, we try to spread out the Managers so they are each in their own Availability Zone. Ideally 3 managers, on 3 different Availability Zones. If we lose an Availability Zone, we only lose one manager, and we still have a quorum, so there are no downtime issues.\nThe problem is that half of the AWS regions have 2 AZ\u0026rsquo;s the other half have 3 or more. Originally we setup our CloudFormation for the least common denominator, which was 2. This wasn\u0026rsquo;t ideal, because that means on a 3 manager swarm, we have two managers on one AZ and one on another AZ. If we lose the AZ with two managers, we lose quorum and Docker Swarm will have issues.\nSo we had a few different options:\nAsk AWS to make sure all regions have at least 3 Availability Zones. Only support the regions that have 3 Availability Zones or more. Have more than one template, one for 2 AZ regions, and one for 3 or more AZ regions. Some how get the CloudFormation template to work with both 3 and 2 AZ regions. Number one wasn\u0026rsquo;t going to happen anytime soon, even if they agreed, it would take a while to implement. So that option was out. Number two would leave off half of the AWS regions, and people wouldn\u0026rsquo;t like that. Number three would be a pain to maintain, and would cause unneeded confusion for people.\nThat left number four, which seems easy at first, unless you have ever used CloudFormation before. CloudFormation is good, but it is very limited and doesn\u0026rsquo;t offer much in the ways of dynamic logic or functionality. After racking my brain for a few days, I decided to think out of the box, and I came up with a solution that handles our requirements, and isn\u0026rsquo;t too much of a hack.\nHow it works.\nI created a new Mapping called AWSRegion2AZ, this is a hard coded mapping that lists the region, along with the Name, number of AZ\u0026rsquo;s and three variables. AZ0, AZ1, and AZ2. Here is what it looks like.\n\u0026#34;AWSRegion2AZ\u0026#34;: { \u0026#34;ap-northeast-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Tokyo\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;ap-northeast-2\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Seoul\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;ap-south-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Mumbai\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;ap-southeast-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Singapore\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;ap-southeast-2\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Sydney\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;3\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; }, \u0026#34;ca-central-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Central\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;eu-central-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Frankfurt\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;eu-west-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Ireland\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;3\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; }, \u0026#34;eu-west-2\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;London\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;sa-east-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Sao Paulo\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;3\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; }, \u0026#34;us-east-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;N. Virgina\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;4\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; }, \u0026#34;us-east-2\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Ohio\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;3\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; }, \u0026#34;us-west-1\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;N. California\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;2\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;0\u0026#34; }, \u0026#34;us-west-2\u0026#34;: { \u0026#34;Name\u0026#34;: \u0026#34;Oregon\u0026#34;, \u0026#34;NumAZs\u0026#34;: \u0026#34;3\u0026#34;, \u0026#34;AZ0\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;AZ1\u0026#34;: \u0026#34;1\u0026#34;, \u0026#34;AZ2\u0026#34;: \u0026#34;2\u0026#34; } } The Name and number of AZ\u0026rsquo;s is self explanatory, so lets me jump to AZ0, AZ1, and AZ2. They are index references for the GetAZs function. For 2 AZ regions, we will have values of 0,1,0 which will reuse the first AZ. For 3 or more AZ regions, we will have values of 0,1,2. Since we have 3 AZs we can use all three.\nWhen you create a Subnet inside of a VPC you do something like this:\n\u0026#34;PubSubnetAz1\u0026#34; : { \u0026#34;DependsOn\u0026#34; : \u0026#34;Vpc\u0026#34;, \u0026#34;Type\u0026#34; : \u0026#34;AWS::EC2::Subnet\u0026#34;, \u0026#34;Properties\u0026#34; : { \u0026#34;VpcId\u0026#34; : { \u0026#34;Ref\u0026#34; : \u0026#34;Vpc\u0026#34; }, \u0026#34;CidrBlock\u0026#34; : { \u0026#34;Fn::FindInMap\u0026#34; : [ \u0026#34;VpcCidrs\u0026#34;, \u0026#34;pubsubnet1\u0026#34;, \u0026#34;cidr\u0026#34; ] }, \u0026#34;AvailabilityZone\u0026#34; : { \u0026#34;Fn::Select\u0026#34; : [ {\u0026#34;Fn::FindInMap\u0026#34;: [\u0026#34;AWSRegion2AZ\u0026#34;, { \u0026#34;Ref\u0026#34;: \u0026#34;AWS::Region\u0026#34; }, \u0026#34;AZ0\u0026#34;]}, { \u0026#34;Fn::GetAZs\u0026#34; : { \u0026#34;Ref\u0026#34; : \u0026#34;AWS::Region\u0026#34; } } ] }, \u0026#34;Tags\u0026#34;: [ { \u0026#34;Key\u0026#34; : \u0026#34;Name\u0026#34;, \u0026#34;Value\u0026#34; : { \u0026#34;Fn::Join\u0026#34;: [ \u0026#34;-\u0026#34;, [ { \u0026#34;Ref\u0026#34;: \u0026#34;AWS::StackName\u0026#34;}, \u0026#34;Subnet1\u0026#34;] ] } } ] } }, This takes a number of parameters but the one we care about is the AvailabilityZone one. We create 3 of these subnets no matter way, even if regions only have 2 availability zones. (Basically the same as above, but only minor changes) When we have a region with only two AZ\u0026rsquo;s there will be 3 subnets, 2 in the first AZ, 1 in the second. When we have a region with 3 or more AZ\u0026rsquo;s there will be a subnet for each of the first 3 AZ\u0026rsquo;s.\nWe assign the AZ by using this function.\n{ \u0026#34;Fn::Select\u0026#34; : [ {\u0026#34;Fn::FindInMap\u0026#34;: [\u0026#34;AWSRegion2AZ\u0026#34;, { \u0026#34;Ref\u0026#34;: \u0026#34;AWS::Region\u0026#34; }, \u0026#34;AZ0\u0026#34;]}, { \u0026#34;Fn::GetAZs\u0026#34; : { \u0026#34;Ref\u0026#34; : \u0026#34;AWS::Region\u0026#34; } } ] } It is a little much to look at, so lets break it up in two smaller parts. First thing it does is gets the list of AZs for the given region. It returns something like this. [\u0026lsquo;us-east-1a\u0026rsquo;, \u0026lsquo;us-east-1b\u0026rsquo;, \u0026lsquo;us-east-1c\u0026rsquo;, \u0026lsquo;us-east-1d\u0026rsquo;]\n{ \u0026#34;Fn::GetAZs\u0026#34; : { \u0026#34;Ref\u0026#34; : \u0026#34;AWS::Region\u0026#34; } } Now given the region we are in, lets assume us-east-1, we want to get the \u0026quot;AWSRegion2AZ\u0026quot;.\u0026quot;us-east-1\u0026quot;.\u0026quot;AZ0\u0026quot; which gives us \u0026ldquo;0\u0026rdquo;\n{\u0026#34;Fn::FindInMap\u0026#34;: [\u0026#34;AWSRegion2AZ\u0026#34;, { \u0026#34;Ref\u0026#34;: \u0026#34;AWS::Region\u0026#34; }, \u0026#34;AZ0\u0026#34;]}, That gives us the following: which means give us the first element from the AZ list, which is \u0026lsquo;us-east-1a\u0026rsquo;\n{ \u0026#34;Fn::Select\u0026#34; : [ \u0026#34;0\u0026#34;, \u0026#34;[\u0026#39;us-east-1a\u0026#39;, \u0026#39;us-east-1b\u0026#39;, \u0026#39;us-east-1c\u0026#39;, \u0026#39;us-east-1d\u0026#39;]\u0026#34; ] } This doesn\u0026rsquo;t matter much for the first 2 subnets, but it does matter on the 3rd one. It will either select the first AZ (for 2 AZ regions), or it will select the 3rd AZ for Regions with 3 or more.\nThis allows us to dynamically put subnets in either 2 or 3 AZ regions. You might be thinking, so what, how does this help me? Well the reason why it matters, is because when you are setting up an AutoScalingGroup, you decide where the servers are put by giving it a list of subnets to shuffle through when placing them. Since subnets are in an AZ, this is how they get spread evenly across AZs.\nHere is a snippet of the AutoScalingGroup to show how you use either 2 or 3 subnets. This is using a conditional called HasOnly2AZs, which is listed below. If HasOnly2AZs is true, then it uses the 2 subnet list, or else it uses the 3.\n\u0026#34;Type\u0026#34; : \u0026#34;AWS::AutoScaling::AutoScalingGroup\u0026#34;, \u0026#34;Properties\u0026#34; : { \u0026#34;VPCZoneIdentifier\u0026#34; : [{ \u0026#34;Fn::If\u0026#34;: [ \u0026#34;HasOnly2AZs\u0026#34;, { \u0026#34;Fn::Join\u0026#34; : [\u0026#34;,\u0026#34;, [ { \u0026#34;Ref\u0026#34; : \u0026#34;PubSubnetAz1\u0026#34; }, { \u0026#34;Ref\u0026#34; : \u0026#34;PubSubnetAz2\u0026#34; } ] ] }, { \u0026#34;Fn::Join\u0026#34; : [\u0026#34;,\u0026#34;, [ { \u0026#34;Ref\u0026#34; : \u0026#34;PubSubnetAz1\u0026#34; }, { \u0026#34;Ref\u0026#34; : \u0026#34;PubSubnetAz2\u0026#34; }, { \u0026#34;Ref\u0026#34; : \u0026#34;PubSubnetAz3\u0026#34; } ] ] } ] }], } Here is the HasOnly2AZs Condition. It uses the NumAZs property of the AWSRegion2AZ mapping, that we talked about above. If the given region, has a NumAZs value of 2, then it is True. Else it is false.\n\u0026#34;Conditions\u0026#34; : { \u0026#34;HasOnly2AZs\u0026#34; : {\u0026#34;Fn::Equals\u0026#34; : [{\u0026#34;Fn::FindInMap\u0026#34;: [\u0026#34;AWSRegion2AZ\u0026#34;, { \u0026#34;Ref\u0026#34;: \u0026#34;AWS::Region\u0026#34; }, \u0026#34;NumAZs\u0026#34;]}, \u0026#34;2\u0026#34;]} }, Now that we have this, we have have one CloudFormation template that will use as many AZs as available. We can recommend that people only use regions with 3 AZs, but if they decide they need to use a region that only has 2 AZs then they can still down.\nThe downsides to this approach is that we have hard coded the list of regions and AZs into the CloudFormation template, when they add a new region, or a new AZ, we will need to also update the template. Considering this doesn\u0026rsquo;t happen often, and there is usually stuff we need to do anyway when a new region goes online (copy AMIs to the new region), this is a low risk.\nThat is all that I have for now, I hope it was helpful. I\u0026rsquo;ll try and keep this page updated. If you have something you think I should add, send me a message on twitter @KenCochrane.\nUpdate: I have added a new blog post on how to use a custom lambda function to get your Availability Zone list.\n","date":"December 16, 2016","externalUrl":null,"permalink":"/2016/12/16/understanding-aws-regions-and-availability-zones-in-cloudformation/","section":"Posts","summary":"","title":"Understanding AWS Regions and Availability Zones in CloudFormation","type":"post"},{"content":"For the past 9 months I have been working on Docker for AWS, which is an easy way to go from nothing, to a Docker Swarm on AWS, in about 10 minutes. The goals of this project were to make it easy for someone with an AWS account to start using Docker Swarm without needing to know anything about Docker and very limited knowledge of AWS.\nWhen I first started, I had no idea what I was going to build, or how I was going to do it. I spent the first 6 weeks, prototyping different ideas, and gave weekly demos to a group of people at Docker. I would take the feedback from those demos and put those towards next weeks prototype. Some of the ideas stuck, some left behind, others were put on list to be added sometime in the future.\nEventually we settled on something that worked well for what we were trying to do, and have a happy balance of form and function. We use CloudFormation to launch the stack, and Docker under the covers to manage the Swarm once it is up and running. It may seem simple, but it is far from it, there is a lot of stuff going on behind the scenes to make it work, and I hope to talk more about that in future posts.\nWe originally launched our private beta at DockerCon US 2016, and since then, we have updated 13 times. We are now in Public Beta, and anyone with an AWS account can try it out. We will hopefully graduate out of Beta, and into General Availability (GA) in the near future. Even though Docker for AWS is constantly evolving, the goals remain the same.\nIf you spin up a 5 node t2.micro cluster and run for under an hour it will cost you less than $0.10. So what are you waiting for, give it a spin, and let me know what you think.\n","date":"December 15, 2016","externalUrl":null,"permalink":"/2016/12/15/my-work-on-docker-for-aws/","section":"Posts","summary":"","title":"My work on Docker for AWS","type":"post"},{"content":"","date":"November 20, 2016","externalUrl":null,"permalink":"/tags/blog/","section":"Tags","summary":"","title":"Blog","type":"tags"},{"content":"","date":"November 20, 2016","externalUrl":null,"permalink":"/tags/hosting/","section":"Tags","summary":"","title":"Hosting","type":"tags"},{"content":"","date":"November 20, 2016","externalUrl":null,"permalink":"/tags/hugo/","section":"Tags","summary":"","title":"Hugo","type":"tags"},{"content":"About a year ago, I was frustrated with my lack of new posts on my blog and decided I needed to fix that. After a little bit of soul searching, I decided to reboot the blog, and start fresh with a new platform. Here are a few of the reasons why I decided to change my blog.\nI didn\u0026rsquo;t like the look of my blog, and there were no good themes available on my old platform. I\u0026rsquo;m not a designer, so the best I\u0026rsquo;m going to get is a theme that is built into the blog platform.\nMy current workflow for writing a post, wasn\u0026rsquo;t very straightforward, and I always forgot the steps needed to publish a new post, and caused me to relearn how to do it every time. This little bit of friction caused my to not to write as much.\nI was hosting on github, which is great, but prevented me from serving over HTTPS.\nBlog platform # The first thing I decided to do, was look at the blogging tool I was using. I did a lot of research on all of the available static site generators that are available, and I settled on hugo. Here are some of the reason why I picked Hugo, over the other available options.\nHugo is written by [Steve Francia] (https://github.com/spf13) who used to work at [Docker] (https://www.docker.com).\nIt is a simple Golang application, and I have been writing more and more Golang recently, so I figured it would give me another way of learning Go.\nIt is really fast, and simple to use. It has a single binary, with a few commands, that are easy to pick up. It also include a dev server, so that you can see what your blog looks like before you publish.\nIt generates a fully static site, so I can easily deploy to any web host that I want.\nIt has a nice collection of themes.\nBlog hosting # Now that I had my blogging tool, I needed a place to host my blog. I wanted something that was cheap, FREE if possible, that allowed, me to also host over HTTPS. I originally looked at hosting on AWS S3, with Cloudfront, but there didn\u0026rsquo;t look like a good way to expire the cache when I made a new post.\nI then looked at a bunch of other platforms, and I decided on using netlify.com to host my blog. Here are some of the reasons why I picked netlify.com:\nThey offered exactly what I was looking for, feature wise.\nCDN Continuous Deployment from github commits HTTPS support via Let\u0026rsquo;s Encrypt. They offered a FREE plan, with a nice set of paid plans that were reasonably priced.\nTheir CTO is [David Calavera] (https://github.com/calavera) who I worked with, when he worked at Docker, and is a super nice guy, who is also super smart, so if he is the CTO, I have full confidence in their platform.\nThey are using Docker to power their platform, and since I love Docker, I do anything I can to support companies in the Docker ecosystem.\nThey love open source, and they have a bunch of cool open source projects.\nConversion # Once I had my new blog platform and hosting figured out, I needed to convert from my old system to my new system. This what took the longest time. My old blog was using the ReStructuredText format, and Hugo supported Markdown. I needed to convert all of the blog posts from one format to the other. Pandoc to the rescue. After a little bit of playing around, I was able to bulk convert about 90% of the documents using Pandoc. The last 10% I had to do by hand, which wasn\u0026rsquo;t too bad.\nOnce the formats were converted, I still needed to move over the metadata since they were different, and I also needed to make sure my urls didn\u0026rsquo;t break when I moved them over. This is what took my a while, and thus the large delay. When I got up enough energy I would work on it, usually when I was bored, and stuck on a cross country flight, with nothing else to do.\nNow with everything converted, I needed to pick a theme. I decided on hugo-icarus. I had to make a few changes, and got everything up and running, and it looked nice locally. Next step, moving it online.\nI signed up at netlify and created a new site that linked to my new git repo on github that had my converted blog files in it. This automatically started a build, which was cool, but it failed. It failed because of the way I had linked my hugo themes, they were setup as submodules, and I guess I did it wrong. I followed their using Hugo on netlify blog post, and this helped me fix the issue. Once that was fixed, I was able to get it to build and deploy quickly, and BAM, my new blog post was live on their temp URL. I was able to play around with it, and make sure everything worked before I made any DNS changes.\nNext step was to change my DNS settings to point to their servers, and enable HTTPS support.\nAll and all, it was super easy, and I\u0026rsquo;m really impressed, Netlify has done a great job, and I highly recommend them, if you are looking for a static website host.\nNew workflow # Now that I have everything setup, here is my new workflow for writing a blog post.\nCreate a new markdown file, add the appropriate meta tags, and write the post.\nRun $ hugo server and preview the post to make sure it looks good.\ngit add/commit the new blog post, and push to github.\nNetlify, takes over from there. They get a webhook from github saying I made a change. They pull down my changes, build the site, push it up to the server, and serve it. They handle any CDN expiration, etc.\nAs you can see, I have reduced all of the friction that was stopping me from writing, in the past. Now I just need to write the post, and commit it, the rest is automated. This will hopefully get me to write more blog posts.\n","date":"November 20, 2016","externalUrl":null,"permalink":"/2016/11/20/i-rebuilt-my-blog-with-hugo-and-moved-to-netlify/","section":"Posts","summary":"","title":"I rebuilt my blog with hugo, and moved to netlify","type":"post"},{"content":"","date":"November 20, 2016","externalUrl":null,"permalink":"/tags/netlify/","section":"Tags","summary":"","title":"Netlify","type":"tags"},{"content":"","date":"February 15, 2015","externalUrl":null,"permalink":"/tags/defender/","section":"Tags","summary":"","title":"Defender","type":"tags"},{"content":"","date":"February 15, 2015","externalUrl":null,"permalink":"/tags/django/","section":"Tags","summary":"","title":"Django","type":"tags"},{"content":"Normally websites do very few logins, someone logs in once and their session is good for a bunch of hours. Since it\u0026rsquo;s a one time thing, it doesn\u0026rsquo;t matter if it isn\u0026rsquo;t very fast. At Docker our authentication system handles requests for both the Docker Hub, as well as all Docker Engine commands that interact with the Docker Hub (docker push, pull, etc). This Authentication system handles thousands of login attempts every minute, so any slow down in the login process has a large impact on our system.\nWe are always looking at ways to improve the security of our systems, and one of the things we looked at implementing was a way to prevent brute force login attempts. The first thing we did was look to see if there was an open source solution available that would do what we needed. The Docker Hub along with the authentication system is written in Python using Django . So we did some research to see what was the best solution available to us. We quickly came to django-axes, which is a great library, and had everything we were looking for. Best of all, it wasn\u0026rsquo;t that hard to add to our system.\nWhen we started using django-axes our traffic was pretty normal, but with the explosive growth of Docker we have seen a huge increases in traffic. As the traffic increased, we started noticing that login times were slowing down. Upon further inspection we noticed that with django-axes turned on it was adding a 200 to 300ms overheard compared to when it was turned off. We did some more digging to see what was causing all of the delay, and we found out the reason is because django-axes stores all of its information in the database and in order to determine if someone is blocked you need to do a bunch of queries, and the more login attempts, the larger the axes database tables get, and thus the queries get slower, and in turn slows down the login process.\nOur first attempts to speed things up was to add more database indexes, and to run a script that kept the axes database tables small. This worked for a little while, but as the traffic increased, it stopped working. It was quickly determined that we would need to replace django-axes. We looked around at other brute force libraries, and there was nothing that offered what we needed for features, was fast, and well maintained.\nSince we couldn\u0026rsquo;t find anything, I started working on a replacement. On one of my many cross country flights, I started working on a replacement. In order to make the process quicker, it started as a fork of django-axes, where I removed the stuff we didn\u0026rsquo;t need, and then replaced the slow parts with faster ones. The main goal was to avoid hitting the database in order to determine if the user was blocked. Since we are big fans of Redis, and we were using it in other places already, we decided to use Redis as our backend for storing all of our data for determining if someone is blocked. We also allow the logging of login attempts to the database, but to speed things up, we defer this to a background celery task.\nAfter a few cross country flights, and some help from some of my co-workers, we now have a library, that we are using on the Docker Hub. We have been using it for about a month now, and the results have been great. With this new library we are seeing under a 10ms impact on our logins, which is really great. We are able to have the features we need without the overhead.\nNow that we have proven the project to be successful, we have decided to open source the library, so that others can also use it, and contribute back any features or improvements they might find important.\nSo without further ado, I\u0026rsquo;m proud to introduce Django Defender, a brute force login preventions library built for speed.\nWe have labeled the first version 0.1, but it is very stable and already production ready. We have very good code coverage (95%+) and we have tests and support for a number of different Python and Django versions.\nWe also have django admin pages that can be used to manage the blocked users and IP addresses.\nPlease try it out, and let us know if you have any questions.\nLinks: # Source code: https://github.com/kencochrane/django-defender PyPi: https://pypi.python.org/pypi/django-defender Version Support: # Python: 2.6.x, 2.7.x, 3.3.x, 3.4.x, PyPy Django: 1.6.x, 1.7.x What\u0026rsquo;s in 0.1: # Configurable: When to block, What to do when blocked. Uses Redis for data store. Blocking IP and usernames when too many login attempts Logging of access attempts to database. Celery for writing access attempt logs to the database in the background Admin pages (integrated with Django admin), to manage the blocked IP\u0026rsquo;s and usernames. How to install: # $ pip install django-defender Admin Screen shots: # ","date":"February 15, 2015","externalUrl":null,"permalink":"/blog/2015/02/introducing-django-defender/","section":"Posts","summary":"","title":"Introducing Django Defender","type":"post"},{"content":"","date":"February 15, 2015","externalUrl":null,"permalink":"/tags/python/","section":"Tags","summary":"","title":"Python","type":"tags"},{"content":"","date":"February 15, 2015","externalUrl":null,"permalink":"/tags/redis/","section":"Tags","summary":"","title":"Redis","type":"tags"},{"content":"","date":"February 14, 2015","externalUrl":null,"permalink":"/tags/api/","section":"Tags","summary":"","title":"Api","type":"tags"},{"content":"","date":"February 14, 2015","externalUrl":null,"permalink":"/tags/facebook/","section":"Tags","summary":"","title":"Facebook","type":"tags"},{"content":"","date":"February 14, 2015","externalUrl":null,"permalink":"/tags/requests/","section":"Tags","summary":"","title":"Requests","type":"tags"},{"content":"","date":"February 14, 2015","externalUrl":null,"permalink":"/tags/s3/","section":"Tags","summary":"","title":"S3","type":"tags"},{"content":"If you have some video files stored in Amazon S3 and you want to upload those videos to a Facebook page, using their video API here is some python code that I used recently.\nI spent a good chunk of a day trying to get this too work, so I\u0026rsquo;m posting this here to help anyone else who is trying to do the same.\nThis code isn\u0026rsquo;t using any special facebook libraries it is just using normal python along with the requests library. It should be a good enough example to figure out how to do most things.\nIt is important to note that you will need a valid access token with correct permissions in order to get this to work, and that isn\u0026rsquo;t covered in this blog post.\nAnother thing that is important to note, that this uses a different API host then the rest of the facebook API. Instead of graph.facebook.com it uses graph-video.facebook.com\nFacebook Video API: https://developers.facebook.com/docs/graph-api/reference/v2.2/user/videos/ Facebook Access Tokens: https://developers.facebook.com/docs/facebook-login/access-tokens Requirements: # $ pip install requests requests-toolbelt Code: # import os import base64 import requests from requests_toolbelt import MultipartEncoder import uuid import logging log = logging.getLogger(__name__) def download_file_to_tmp(source_url): \u0026#34;\u0026#34;\u0026#34; download `source_url` to /tmp return the full path, doing it in chunks so that we don\u0026#39;t have to store everything in memory. \u0026#34;\u0026#34;\u0026#34; log.debug(\u0026#34;download {0}\u0026#34;.format(source_url)) tmp_location = \u0026#34;/tmp/s3_downloads\u0026#34; # come up with a random name to avoid clashes. rand_name = str(uuid.uuid4().get_hex().lower()[0:6]) local_filename = source_url.split(\u0026#39;/\u0026#39;)[-1] # get the extension if it has one if local_filename.count(\u0026#34;.\u0026#34;) \u0026gt; 0: ext = local_filename.split(\u0026#39;.\u0026#39;)[-1] tmp_filename = u\u0026#34;{0}.{1}\u0026#34;.format(rand_name, ext) else: tmp_filename = u\u0026#34;{0}.mp4\u0026#34;.format(local_filename) temp_media_location = os.path.join(tmp_location, tmp_filename) # make the temp directory if not os.path.exists(tmp_location): os.makedirs(tmp_location) r = requests.get(source_url, stream=True) log.debug(\u0026#34;headers = {0}\u0026#34;.format(r.headers)) with open(temp_media_location, \u0026#39;wb\u0026#39;) as f: for chunk in r.iter_content(chunk_size=1024): if chunk: # filter out keep-alive new chunks f.write(chunk) f.flush() os.fsync(f.fileno()) log.debug(\u0026#34;finished download to {0}\u0026#34;.format(temp_media_location)) return temp_media_location def remove_file(temp_file): \u0026#34;\u0026#34;\u0026#34; Given a valid file path remove it \u0026#34;\u0026#34;\u0026#34; if os.path.exists(temp_file): os.remove(temp_file) def upload_file(video_url, page_id, poster_url, access_token, description, title): \u0026#34;\u0026#34;\u0026#34; ``video_url``: this is where the video is in s3. ``page_id``: me or a page_id for the page you want to post too. ``poster_url``: the url to the poster (thumbnail) for this video ``access_token``: your facebook access token with permissions to upload to the page you want to post too. ``description``: the description of the video you are posting. ``title``: the title of the video you are posting \u0026#34;\u0026#34;\u0026#34; # download to data local_video_file = download_file_to_tmp(video_url) video_file_name = local_video_file.split(\u0026#34;/\u0026#34;)[-1] if video_file_name and video_file_name.count(\u0026#34;.\u0026#34;) == 0: log.debug(\u0026#34;video_file_name has no ext {0}\u0026#34;.format(video_file_name)) # if it doesn\u0026#39;t have an extension add one to it. video_file_name = \u0026#34;{0}.mp4\u0026#34;.format(video_file_name) log.debug(\u0026#34;video_file_name converted to {0}\u0026#34;.format(video_file_name)) # download to data local_poster_file = download_file_to_tmp(poster_url) # need to encode it. with open(local_poster_file, \u0026#34;rb\u0026#34;) as image_file: poster_encoded_string = base64.b64encode(image_file.read()) # need binary rep of this, not sure if this would do it # put it all together to post to facebook if page_id or page_id == \u0026#39;me\u0026#39;: path = \u0026#39;me/videos\u0026#39; else: path = \u0026#34;{0}/videos\u0026#34;.format(page_id) fb_url = \u0026#34;https://graph-video.facebook.com/{0}?access_token={1}\u0026#34;.format( path, access_token) log.debug(\u0026#34;video_file = {0}\u0026#34;.format(local_video_file)) log.debug(\u0026#34;thumb_file = {0}\u0026#34;.format(local_poster_file)) log.debug(\u0026#34;start upload to facebook\u0026#34;) # multipart chunked uploads m = MultipartEncoder( fields={\u0026#39;description\u0026#39;: description, \u0026#39;title\u0026#39;: title, \u0026#39;thumb\u0026#39;: poster_encoded_string, \u0026#39;source\u0026#39;: (video_file_name, open(local_video_file, \u0026#39;rb\u0026#39;))} ) r = requests.post(fb_url, headers={\u0026#39;Content-Type\u0026#39;: m.content_type}, data=m) if r.status_code == 200: j_res = r.json() facebook_video_id = j_res.get(\u0026#39;id\u0026#39;) log.debug(\u0026#34;facebook_video_id = {0}\u0026#34;.format(facebook_video_id)) else: log.error(\u0026#34;Facebook upload error: {0}\u0026#34;.format(r.text)) # delete the tmp files remove_file(local_video_file) remove_file(local_poster_file) return facebook_video_id ","date":"February 14, 2015","externalUrl":null,"permalink":"/blog/2015/02/upload-video-to-facebook-from-s3-using-python/","section":"Posts","summary":"","title":"Uploading a Video to Facebook from AWS S3 using python","type":"post"},{"content":"","date":"February 14, 2015","externalUrl":null,"permalink":"/tags/video/","section":"Tags","summary":"","title":"Video","type":"tags"},{"content":"","date":"August 23, 2013","externalUrl":null,"permalink":"/tags/guidebook/","section":"Tags","summary":"","title":"Guidebook","type":"tags"},{"content":" Introduction # The goal of this tutorial is to introduce you to Docker, show you what it can do, and how to get it up and running on your system, and how to use it to make your life better.\nThis guide is open source and available on github.com. If you would like to add to it or fix something, please fork it and submit a pull request.\nTable of Contents\nWhat is Docker? # Docker is a tool created by the folks at dotCloud to make using LinuX Containers (LXC) easier to use. Linux Containers are basically light weight Virtual Machines (VM). A linux container runs Unix processes with strong guarantees of isolation across servers. Your software runs repeatably everywhere because its Container includes all of its dependencies.\nIf you still don\u0026rsquo;t understand what Docker is, and what it can do for you, don\u0026rsquo;t worry, keep reading and it will become clear soon enough.\nHow are Docker\u0026rsquo;s Containers Different from Virtual Machines? # Docker, which uses LinuX Containers (LXC) run in the same kernel as it\u0026rsquo;s host. This allows it to share a lot of the host\u0026rsquo;s resources. It also uses AuFS for the file system. It also manages the networking for you as well.\nAuFS is a layered file system, so you can have a read only part, and a write part, and it merges those together. So you could have the common parts of the file system as read only, which are shared amongst all of your containers, and then give each container it\u0026rsquo;s own mount for writing.\nSo let\u0026rsquo;s say you have a container image that is 1GB in size. If you wanted to use a Full VM, you would need to have 1GB times x number of VMs you want. With LXC and AuFS you can share the bulk of the 1GB and if you have 1000 containers you still might only have a little over 1GB of space for the containers OS, assuming they are all running the same OS image.\nA full virtualized system gets it\u0026rsquo;s own set of resources allocated to it, and does minimal sharing. You get more isolation, but it is much heavier (requires more resources).\nWith LXC you get less isolation, but they are more lightweight and require less resources. So you could easily run 1000\u0026rsquo;s on a host, and it doesn\u0026rsquo;t even blink. Try doing that with Xen, and unless you have a really big host, I don\u0026rsquo;t think it is possible.\nA full virtualized system usually takes minutes to start, LXC containers take seconds, and most times less then a second.\nThere are pros and cons for each type of virtualized system. If you want full isolation with guaranteed resources then a full VM is the way to go. If you just want to isolate processes from each other and want to run a ton of them on a reasonably sized host, then LXC might be the way to go.\nFor more information check out these set of blog posts which do a good job of explaining now LXC works: http://blog.dotcloud.com/under-the-hood-linux-kernels-on-dotcloud-part\nInstalling Docker # Before you can install Docker you need to decide how you want to install it. There are three ways to install it, you can install from source, download a compiled binary, or install via your systems package manager.\nFor detailed instructions on how to install Docker on your system for each of the following steps, check out the official Docker documentation http://docs.docker.io/en/latest/installation/\nRequirements # In order for Docker to run correctly on your server, you need to have a few things. For more details on the kernel requirements see this page: see http://docs.docker.io/en/latest/installation/kernel/\nKernel version greater then 3.8 and Cgroups and namespaces must be enabled. AUFS : AUFS is included in the kernels built by the Debian and Ubuntu distributions, but not built into the standard kernel, so if you are using another distribution you will need to add it to your kernel. LXC : This is most likely already installed on your system and kernel, you might just need to install a system package or two. See the install instructions for your distribution to get a list of packages. Kernel version # The reason why Docker needs to run in a kernel version of 3.8 or greater is because there are some kernel bugs that are in the older versions that cause problems in some cases. Some people have ran Docker fine on lower kernels, so if you can\u0026rsquo;t run on 3.8, do so at your own risk. There is talk about an effort to back port the bug fixes to the older kernel trees, so that in the future they will be available on the older kernel versions. For more information about this see. https://github.com/dotcloud/docker/pull/1062\nAUFS # Currently AUFS is the standard file system for Docker, but there is an effort underway to make the filesystem more pluggable, so that we can use different file systems with Docker. AUFS will most likely not be available in future Ubuntu releases, and UnionFS doesn\u0026rsquo;t look like it will be getting added to the kernel anytime soon, so we can\u0026rsquo;t add that as a replacement. The current replacement looks like BTRFS.\nPackage Manager # The most common way to install Docker is via your server\u0026rsquo;s package manager. On Ubuntu that is as simple as running the following command sudo apt-get install lxc-docker. This is an easy way to install docker, and keep it up to date.\nThe package will also install an init script so that the docker daemon will start up automatically.\nIf you are installing on a production server, this is the recommended way to install.\nUpgrading: # To upgrade you would upgrade the same way you upgrade any other package for your system. On Ubuntu you would run \u0026lsquo;sudo apt-get upgrade\u0026rsquo;\nBinaries # If a docker package isn\u0026rsquo;t available for your package manager, you can download the binaries directly. When a new version of docker is released the binaries are uploaded to http://get.docker.io, so that you can download directly from there. Here is an example on how to download the latest docker release.\nwget http://get.docker.io/builds/Linux/x86_64/docker-latest.tgz tar -xf docker-latest.tgz This just downloads the docker binary, to get it to run you would still need to put the binary in a good location, and create an init script so that it will start on system reboots.\nInit script examples: # Debian init: https://github.com/dotcloud/docker/blob/master/packaging/debian/lxc-docker.init Ubuntu Upstart: https://github.com/dotcloud/docker/blob/master/packaging/ubuntu/docker.upstart Upgrading: # To upgrade you would need to download the latest version, make a backup of the current docker binary, replace the current one with the new one, and restart your daemon. The init script should be able to stay the same.\nMore information: # http://docs.docker.io/en/latest/installation/binaries/\nFrom Source # Installing from a package manager or from a binary is fine if you want to only install released versions. But if you want to be on the cutting edge and install some features that are either on a feature branch, or something that isn\u0026rsquo;t released yet, you will need to compile from source.\nCompiling from source is a little more complicated because you will need to have GO 1.1 and all other dependences install on your system, but it isn\u0026rsquo;t too bad.\nHere is what you need to do to get it up and running on Ubuntu:\nsudo apt-get install python-software-properties sudo add-apt-repository ppa:gophers/go sudo apt-get update sudo apt-get -y install lxc xz-utils curl golang-stable git aufs-tools export GOPATH=~/go/ export PATH=$GOPATH/bin:$PATH mkdir -p $GOPATH/src/github.com/dotcloud cd $GOPATH/src/github.com/dotcloud git clone git://github.com/dotcloud/docker.git cd docker go get -v github.com/dotcloud/docker/... go install -v github.com/dotcloud/docker/... Then run the docker daemon:\nsudo $GOPATH/bin/docker -d If you make any changes to the code, run the go install command (above) to recompile docker. Feel free to change the git clone command above to your own fork, to make pull request\u0026rsquo;s easier.\nDocker requires Go 1.1, if you have an older version it will not compile correctly.\nDocker Daemon # The Docker daemon needs to be running on your system to control the containers. The daemon needs to be run as Root so that it can have access to everything it needs.\nStarting the daemon # There are two ways to start the daemon, you can start it using an init script so that it starts on system boot, and manually starting the daemon and sending to the background. The init script is the preferred way of doing this. If you install Docker via a package manager you already have the init script on your system.\nTo start it manually you need to use a command like this.\nsudo \u0026lt;path to\u0026gt;/docker -d \u0026amp; When Docker starts, it will listen on 127.0.0.1:4243 to allow only local connections but you can set it to 0.0.0.0:4243 or a specific host ip to give access to everybody.\nTo change the host and port that docker listens to you will need to use the -H flag when starting docker.\n-H accepts host and port assignment in the following format: tcp://[host][:port] or unix://path For example:\ntcp://host -\u0026gt; tcp connection on host:4243 tcp://host:port -\u0026gt; tcp connection on host:port tcp://:port -\u0026gt; tcp connection on 127.0.0.1:port unix://path/to/socket -\u0026gt; unix socket located at path/to/socket When you do this, you need to also let the docker client know what daemon you want to connect too. To do that you have to also pass in the -H flag to with the ip:port of the daemon to connect too.\n# Run docker in daemon mode on port 5555 sudo \u0026lt;path to\u0026gt;/docker -H 0.0.0.0:5555 \u0026amp; # Download a base image using the daemon on port 5555 docker -H :5555 pull base You can use multiple -H, for example, if you want to listen on both tcp and a unix socket\n# Run docker in daemon mode on 127.0.0.1:4243 and unix socket unix:///var/run/docker.sock sudo \u0026lt;path to\u0026gt;/docker -H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock # Download a base image (no need to put the -H since it is listen on default port :4243) docker pull base # OR (pull via the unix socket) docker -H unix:///var/run/docker.sock pull base Configuration # Currently if you want to configure the docker daemon, you can either pass in command switches to the docker daemon on startup, or you can set ENV variables that the docker daemon will pick up. I have proposed a better approach for configuring docker, the idea is to use a docker.conf file so that it is easier to set and is more obvious. Details can be found here: https://github.com/dotcloud/docker/issues/937\nThere are two ENV variables that you can set today, there maybe more added in the future.\nDEBUG # This tells the Docker daemon that you want more debug information in your logs.\ndefaults to DEBUG=0, set to DEBUG=1 to enable.\nDOCKER_INDEX_URL # This tells Docker which Docker index to use. You will most likely not use this setting, it is mostly used for Docker developer when they want to try things out with the test index before they release the code.\ndefaults to DOCKER_INDEX_URL=https://index.docker.io\nExample # This is how you would set it if it was in an init file:\n# /etc/init/docker.conf env LC_ALL=\u0026quot;en_US.UTF-8\u0026quot; env DOCKER_INDEX_URL=\u0026quot;https://index.docker.io\u0026quot; env DEBUG=1 exec /usr/local/bin/docker -d Logs # There is no official Docker log file right now, I have opened an issue and requested one: https://github.com/dotcloud/docker/issues/936 but in the meantime if you are using upstart you can use /var/log/upstart/docker.log which has some information, but not as much as I would like.\nTesting Docker install # Now that you have Docker running, you can start to issue some Docker commands to see how things are working. The very first commands that I always run are Docker version and Docker info. These tell me quickly if I have everything working correctly. :\n$ docker version Client version: 0.4.8 Server version: 0.4.8 Go version: go1.1 $ docker info Containers: 0 Images: 0 WARNING: No memory limit support WARNING: No swap limit support Notice that I have two warnings for my docker info. If you use Debian or Ubuntu kernels, and want to enable memory and swap accounting, you must add the following command-line parameters to your kernel:\ncgroup_enable=memory swapaccount=1 On Debian or Ubuntu systems, if you use the default GRUB bootloader, you can add those parameters by editing /etc/default/grub and extending GRUB_CMDLINE_LINUX. Look for the following line:\nGRUB_CMDLINE_LINUX=\u0026quot;\u0026quot; And replace it by the following one:\nGRUB_CMDLINE_LINUX=\u0026quot;cgroup_enable=memory swapaccount=1\u0026quot; Then run update-grub, and reboot the server.\nTerminology # There are going to be some terms that you hear throughout this tutorial, to make sure you understand what we are talking about, I\u0026rsquo;ll explain a few of them here.\nImage # An image is a read only layer used to build a container. They do not change.\nContainer # Is basically a self contained runtime environment that is built using one or more images. You can commit your changes to a container and create an image.\nindex / registry # These are public or private servers where people can upload their repositories so they can easily share what they made.\nRepository # A repository is a group of images located in the docker registry. There are two types of repositories, Top level and user repositories. Top level repositories don\u0026rsquo;t have a \u0026lsquo;/\u0026rsquo; in the name and they are usually reserved for base images. These Top level repositories is what most people build their repositories on top of. They are controlled by the maintainers of Docker. User repositories are repositories that anyone can upload into the registry and share with other people.\nGetting Help with Docker # If you have a question or problem when using Docker, there are a number of different ways to help you. Here is a list of the ways, pick the one that works best for you.\nIRC: #docker on freenode, There are a bunch (250+) people normally in this channel, come on in, and ask your question, we are very friendly and we don\u0026rsquo;t bite. Also newbie questions are welcome. Email: There is a google group called docker-club. Join the list, and ask any questions you might have. https://groups.google.com/d/forum/docker-club Twitter: http://twitter.com/getdocker/ Follow along, if you aren\u0026rsquo;t already, lots of great info posted every day. StackOverflow: We love Stack Overflow, if you also enjoy it, feel free to post a question using the docker tag, and one of the many Docker fans will get back to you quickly. If you love getting points, feel free to answer questions as well. Bugs and feature requests: If you have a bug or feature request, submit them to GitHub. http://www.github.com/dotcloud/docker Part 1. Getting Started # Now that we have the boring stuff out of the way lets start playing with Docker. The very first example we are going to do is a very simple one, we will spin up a container and print hello world to the screen. :\n#run a simple echo command, that will echo hello world back to the console over standard out. $ docker run base /bin/echo hello world hello world If this was your first docker command you will notice that it will need to download the base image first. It only needs to do this once, and it caches it locally so you don\u0026rsquo;t need to do this again. We could have broken these out into two commands docker pull base and then the docker run command, but I was lazy and put them together, and Docker is smart enough to know what I want to do, and do it for me.\nNow you might be wondering what is Docker doing here exactly. It doesn\u0026rsquo;t look like much because we picked such a simple example, but here is what is happening.\nGenerated a new LXC container Created a new file system Mounted a read/write layer Allocated network interface Setup IP Setup NATing Executed the process in the container Captured it\u0026rsquo;s output Printed to screen Stopped the container All in under a second!\nIf we run the docker images command we should see the base image in our list. :\n$ docker images REPOSITORY TAG ID CREATED SIZE base latest b750fe79269d 3 months ago 24.65 kB (virtual 180.1 MB) base ubuntu-12.10 b750fe79269d 3 months ago 24.65 kB (virtual 180.1 MB) base ubuntu-quantal b750fe79269d 3 months ago 24.65 kB (virtual 180.1 MB) base ubuntu-quantl b750fe79269d 3 months ago 24.65 kB (virtual 180.1 MB) Notice how you see the same image more then once, that is because there are more then one tag for the same image.\nIf we want to see the container we just ran we can run the docker ps command. Since it isn\u0026rsquo;t running anymore we need to use the -a flag to show us all of the image:\n$ docker ps -a ID IMAGE COMMAND CREATED STATUS PORTS 861361e27501 base:latest /bin/echo hello world 1 minutes ago Exit 0 Lets do something a little more complicated. We are going to do the same thing, but instead of having the container exit right after we start, we want it to keep running in the background, and print hello world every second:\n$ CONTAINER_ID=$(docker run -d base /bin/sh -c \u0026quot;while true; do echo hello world; sleep 1; done\u0026quot;) $ echo $CONTAINER_ID f684fc88aec3 $ docker ps ID IMAGE COMMAND CREATED STATUS PORTS f684fc88aec3 base:latest /bin/sh -c while tru 33 seconds ago Up 33 seconds There we go, now lets see what the container is doing by looking at the logs for the container:\n$ docker logs f684fc88aec3 hello world hello world hello world hello world hello world .. (trimmed) Now lets attach to the container and see the results in realtime:\n$ docker attach f684fc88aec3 hello world hello world hello world Ok, enough fun for this container, lets stop it.\n$ docker stop f684fc88aec3 f684fc88aec3\n$ docker ps ID IMAGE COMMAND CREATED STATUS PORTS\nAnother thing we could have done to look at the container was inspect the container, we can do this while it is running or after it stopped:\n$ docker inspect f684fc88aec3 [{ \u0026quot;ID\u0026quot;: \u0026quot;f684fc88aec3bf5b74df2fe03da1fe7cebf07a89d308b6ac7e8a6f14d9c9a3dd\u0026quot;, \u0026quot;Created\u0026quot;: \u0026quot;2013-07-05T21:23:31.27766521Z\u0026quot;, \u0026quot;Path\u0026quot;: \u0026quot;/bin/sh\u0026quot;, \u0026quot;Args\u0026quot;: [ \u0026quot;-c\u0026quot;, \u0026quot;while true; do echo hello world; sleep 1; done\u0026quot; ], \u0026quot;Config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;f684fc88aec3\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: false, \u0026quot;AttachStdout\u0026quot;: false, \u0026quot;AttachStderr\u0026quot;: false, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: false, \u0026quot;OpenStdin\u0026quot;: false, \u0026quot;StdinOnce\u0026quot;: false, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: [ \u0026quot;/bin/sh\u0026quot;, \u0026quot;-c\u0026quot;, \u0026quot;while true; do echo hello world; sleep 1; done\u0026quot; ], \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;base\u0026quot;, \u0026quot;Volumes\u0026quot;: {}, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: [] }, \u0026quot;State\u0026quot;: { \u0026quot;Running\u0026quot;: false, \u0026quot;Pid\u0026quot;: 0, \u0026quot;ExitCode\u0026quot;: 137, \u0026quot;StartedAt\u0026quot;: \u0026quot;2013-07-05T21:23:31.298200635Z\u0026quot;, \u0026quot;Ghost\u0026quot;: false }, \u0026quot;Image\u0026quot;: \u0026quot;b750fe79269d2ec9a3c593ef05b4332b1d1a02a62b4accb2c21d589ff2f5f2dc\u0026quot;, \u0026quot;NetworkSettings\u0026quot;: { \u0026quot;IPAddress\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;IPPrefixLen\u0026quot;: 0, \u0026quot;Gateway\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Bridge\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;PortMapping\u0026quot;: null }, \u0026quot;SysInitPath\u0026quot;: \u0026quot;/usr/bin/docker\u0026quot;, \u0026quot;ResolvConfPath\u0026quot;: \u0026quot;/etc/resolv.conf\u0026quot;, \u0026quot;Volumes\u0026quot;: {}, \u0026quot;VolumesRW\u0026quot;: {} }] There is a lot of information there, you might not need it now, but you may need it in the future, so it is nice to have it available.\nNow that you know the basics go to part 2, and learn how to build an image.\nPart 2. Building an image # Our goal for this part is to create our own Redis server container. The first thing we will need to do is decide which image we want to build on. I usually pick the ubuntu image, but sometimes it is nice to start from something a little higher so that I don\u0026rsquo;t have to recreate steps, and I can build on the shoulders of others.\nWe are going to run /bin/bash with the -i and the -t flags. -i tells Docker to keep stdin open even if not attached, and -t is to allocate a pseudo-tty. Once we run the command, we will be connected into the container, and all commands at this point are running from inside the container. :\n$ docker run -i -t ubuntu /bin/bash root@dda8bfc22397:/# hostname dda8bfc22397 root@dda8bfc22397:/# ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 18060 1940 ? S 21:40 0:00 /bin/bash root 11 0.0 0.0 15532 1136 ? R+ 21:41 0:00 ps aux OK, it looks like we are in, and things are working well, now lets get to work.\nWe are going to update apt and then install redis:\n$ apt-get update $ apt-get install redis-server $ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 18060 1944 ? S 22:21 0:00 /bin/bash redis 116 0.0 0.0 36628 1656 ? Ssl 22:22 0:00 /usr/bin/redis-server /etc/redis/redis.conf root 125 0.0 0.0 15532 1140 ? R+ 22:23 0:00 ps aux $ exit Now we have a container with redis installed. Less see what we did to the container:\n$ docker diff dda8bfc22397 A /.bash_history C /dev A /dev/kmsg C /etc C /etc/bash_completion.d A /etc/bash_completion.d/redis-cli C /etc/default A /etc/default/redis-server .. (trimmed) It should show you what files have changed (C) and which ones were added (A). Lets save our work so we can reuse this in the future. To do this we need to docker commit the container to create an image. In order to commit changes you need your container_id. If you don\u0026rsquo;t remember it don\u0026rsquo;tw worry you can get it from docker ps -a:\n$ docker ps -a # grab the container id (this will be the first one in the list) $ docker commit \u0026lt;container_id\u0026gt; \u0026lt;your username\u0026gt;/redis 82ebf04d9385 It returns an image id. if we run docker images we should see it listed:\n$ docker images REPOSITORY TAG ID CREATED SIZE base latest b750fe79269d 3 months ago 24.65 kB (virtual 180.1 MB) base ubuntu-12.10 b750fe79269d 3 months ago 24.65 kB (virtual 180.1 MB) base ubuntu-quantal b750fe79269d 3 months ago 24.65 kB (virtual 180.1 MB) base ubuntu-quantl b750fe79269d 3 months ago 24.65 kB (virtual 180.1 MB) kencochrane/redis latest 82ebf04d9385 About a minute ago 98.46 MB (virtual 278.6 MB) Lets run our new image and see if it works:\n$ docker run -d -p 6379 kencochrane/redis /usr/bin/redis-server 4cbaae2f67d0 The -d tell docker to run it in the background, just like our Hello World daemon from the last part. -p 6379 says to use 6379 as the port for this container.\nTest 1 Connect to the container with the redis-cli. :\n$ docker ps # grab the new container id $ docker inspect \u0026lt;container_id\u0026gt; | grep IPAddress # grab the ipaddress of the container \u0026quot;IPAddress\u0026quot;: \u0026quot;172.16.42.5\u0026quot;, redis-cli -h 172.16.42.5 -p 6379 redis 10.0.3.32:6379\u0026gt; set docker awesome OK redis 10.0.3.32:6379\u0026gt; get docker \u0026quot;awesome\u0026quot; redis 10.0.3.32:6379\u0026gt; exit Connect to the public IP with the redis-cli. :\n$ docker ps # grab the new container id $ docker port \u0026lt;container_id\u0026gt; 6379 # grab the external port 49153 ip addr show # grab the host ip address redis-cli -h \u0026lt;host ipaddress\u0026gt; -p 49153 redis 192.168.0.1:49153\u0026gt; set docker awesome OK redis 192.168.0.1:49153\u0026gt; get docker \u0026quot;awesome\u0026quot; redis 192.168.0.1:49153\u0026gt; exit We just proved that it is working as it should, we can now stop the container using docker stop. You have now created your first Docker image. Continue on to the next part to learn how to use that image on another host, and share it with the world.\nPart 3: Docker Index/registry # When you create an image it is only available on that server. In the past, if you wanted to use the same image on another server, you would need to recreate the image, which isn\u0026rsquo;t ideal because there is no way to guarantee that the two images are the same. To make moving images around, and sharing them easier, the Docker team created the Docker index.\nThe Docker Index is a public Registry where people can upload their custom images and share them with others. This is also where the base images are located and where you pull from when doing a docker pull. There are two parts to the Docker Index. There is a web component that makes it easier for you to mange your images and account with a graphical interface. There is also the API which is what the Docker client uses to interact with the index. This allows you to do some of the tasks from the command line or the web UI.\nThe Docker Registry is server that stores all of the images and repositories. The Index just has the metadata about the images, repositories and the user accounts, but all of the images and repositories are stored in the Docker Registry.\nCreating an Account on the Docker Index # There are two ways to create an account on the Docker Index. Either way requires that you enter a valid email address and that the email address is confirmed before you can activate the account. So make sure you enter a valid email address, and then check you email after registering so that you can click the confirmation link and confirm the account.\nCommand Line # If you want to register for an account from the command line you can use the docker login command. The Docker login command will either register an account for you, or if you already have an account it will log you into the Index.\nWhen you register via the command line, it will register you and login you in a the same time. Remember to click on the activation link in the confirmation email, or else your account isn\u0026rsquo;t fully active. :\n$ docker login Username (): myusername Password: Email (): myusername@example.com Login Succeeded Web site # If you prefer to register from a web browser, then go to https://index.docker.io/account/signup/ and then fill out the form, and then click on the activation link sent in the confirmation email.\nOnce you are activated, you will still need to login to the Docker Index from your Docker client on your server, so that you can link the two. :\n$ docker login Username (): myusername Password: Email (): myusername@example.com Login Succeeded Credentials # When you login to the Docker Index from the Docker client, it will store your login information, so you don\u0026rsquo;t have to enter it again. Depending on what Docker client version you are using it will either be located at ~/.dockercfg or /var/lib/docker/.dockercfg. If you are having issues logging in you, can delete this file, and it will re-prompt you for your username and password the next time you login. Running Docker login should do the same thing, so do that first, and use this for a last resort.\nSearch # There are a lot of Docker images in the Index, with more getting added everyday. Before you go ahead and create your own, you should see if someone has already created what you wanted. The best way to find images is via the docker search command on the command line, or via the Docker Index website. :\n$ docker search memcache Found 5 results matching your query (\u0026quot;memcache\u0026quot;) NAME DESCRIPTION ehazlett/memcached Memcached 1.4.15. Specify the following e... jbarbier/memcached memcached checkraiser/memcached arcus/memcached bacongobbler/memcached Pulling # When you found an image that you want to pull down and try out, you would use the docker pull command. It will then connect to the Docker Index find the repository that you want, and it will let the Docker client know where in the Docker Registry it can download it. :\n$ docker pull jbarbier/memcached Pushing # If you have a repository that you want to share with someone then you would need to push it into the Docker Index/Registry using the docker push command. When you do a push, it will contact the Docker Index, and make sure you are logged in, have permission to push, and that the same repository doesn\u0026rsquo;t already exist. If everything looks good, it will then return a special authorization token that the Docker client will use when push up the repository to the Docker Registry.\nSince the Docker Register doesn\u0026rsquo;t have any concept of authorization, or user accounts, it relies on Authorization tokens to manage permissions. The nice thing about this, is that Docker hides this all from you, and you don\u0026rsquo;t even need to worry about it, it will just work assuming you have permission to push.\nLet\u0026rsquo;s push the repository that we created in the last part, so that others can use it. :\n$ docker push kencochrane/redis Now that it is up on the registry we can use it on any Docker host, and we just need to do a Docker pull to get it on the host, and I\u0026rsquo;ll know it is going to be the same every time.\nRepository Description # If you want to add a description to your repository so that it lets people know what it does, you can login to the website and edit the description there. There are two descriptions, a short one, which is what shows up in search results, and is plain text. There is also a full description which allows MarkDown and is used to give more detailed information.\nDeleting a Repository # If you made a mistake and need to delete a repository, you can do this by logging into the Docker Index website, and clicking on the repository settings and clicking the delete button. Make sure this is what you want to do, because there is no turning back once you do this.\nPart 4: Docker Buildfiles # TODO:\nGo over what a Docker Buildfile is, and how to make their own. With examples Part 5: Advanced Usage # TODO:\ndocker run : - limiting memory, cpu - detached vs attached - volume/bind mounting More? Part 6: Using a Private Registry # One of the things that makes Docker so useful is how easy it is to pull ready-to-use images from a central location, Docker\u0026rsquo;s Central Registry. It is just as easy to push your own image (or collection of tagged images as a repository) to the same public registry so that everyone can benefit from your newly Dockerized service.\nBut sometimes you can\u0026rsquo;t share your repository with the world because it contains proprietary code or confidential information. Today we are introducing an easy way to share repositories on your own registry so that you can control access to them and still share them among multiple Docker daemons. You can decide if your registry is public or private.\nYou\u0026rsquo;ll need the latest version of Docker (\u0026gt;=0.5.0) to use this new feature, and you must run this version as both the daemon and the client. You\u0026rsquo;ll also need the Docker registry code.\nUsing Push and Pull with a Private Registry # You\u0026rsquo;ve already seen how to push and pull from the Central Registry. To push to or pull from your own registry, you just need to add the registry\u0026rsquo;s location to the repository name. It will look like my.registry.address:port/repositoryname.\nLet\u0026rsquo;s say I want to push the repository \u0026ldquo;ubuntu\u0026rdquo; to my local registry, which runs on my local machine, on the port 5000: :\n# First, make sure you have the \u0026quot;ubuntu\u0026quot; repository: docker pull ubuntu # Then, find the image id that corresponds to the ubuntu repository docker images | grep ubuntu | grep latest ubuntu latest 8dbd9e392a96 12 weeks ago 263 MB (virtual 263 MB) # Almost there! # Tag to create a repository with the full registry location. # The location becomes a permanent part of the repository name. docker tag 8dbd9e392a96 localhost.localdomain:5000/ubuntu # Finally, push the new repository to its home location. docker push localhost.localdomain:5000/ubuntu Obviously, the push will fail if no registry server answer locally on the port 5000. We\u0026rsquo;ll briefly show how to start your own registry server in the next subsection.\nnote\nThe punctuation in the repository name is important! Docker looks for either a \u0026ldquo;.\u0026rdquo; (domain separator) or \u0026ldquo;:\u0026rdquo; (port separator) to learn that the first part of the repository name is a location and not a user name. If you just had localhost without either .localdomain or :5000 (either one would do) then Docker would believe that localhost is a username, as in localhost/ubuntu or samalba/hipache. It would then try to push to the default Central Registry. Having a dot or colon in the first part tells Docker that this name contains a hostname and that it should push to your specified location instead.\nInstalling Your Own Registry # Docker-Registry is a an Open Source Python application available on Github: https://github.com/dotcloud/docker-registry\nYou can use the Docker-Registry to provide a private or public registry service for Docker repositories. Since it is your host, you can control access to it by putting it on a private network or otherwise protecting its service port. You\u0026rsquo;ll want to choose the DNS name of the host carefully, since that name will become a permanent part of each repository\u0026rsquo;s name (e.g. my.registry.name/myrepository).\nYou can test out the Docker-Registry first on your local machine (presuming you have a Python environment set up). :\ngit clone https://github.com/dotcloud/docker-registry.git cd docker-registry cp config_sample.yml config.yml pip install -r requirements.txt gunicorn --access-logfile - --log-level debug --debug \\ -b 0.0.0.0:5000 -w 1 wsgi:application That sets up the Docker-Registry to listen on all your network interfaces on port 5000. You\u0026rsquo;re using the dev flavor configuration by default, which uses local storage for the repositories. The configuration file (config.yml) also allows you to specify other flavors, like production, and to use other storage backends, like S3.\nThere is currently no authentication built into the Docker-Registry, so if you want to keep this private, you\u0026rsquo;ll need to keep the host on a private network. We\u0026rsquo;d recommend running a production Docker-Registry behind an Nginx server which sipplies chunked transfer encoding.\nPart 7: Automating Docker # Running docker commands on the command line are a good way to start, but if you need to automate what you are doing, it isn\u0026rsquo;t ideal. To make this better Docker provides a REST based remote API. The remote API allows you to do everything that the command line does. In fact the command line is just a client for the REST API.\nRemote API # Docker provides a remote API for the docker daemon so that you can control it programmatically, for documentation on how it works check out the Docker Remote API Docs\nDocker Web UI\u0026rsquo;s # Docker is a completly command line experience, which is fine for hackers, but some people prefer a more graphical experience, and for those folks I would recommend checking out these projects that people have started.\nDockland # A ruby based Docker web UI\nCode: https://github.com/dynport/dockland\nShipyard # A python/django based Docker web UI\nCode: https://github.com/ehazlett/shipyard\nDockerUI # An Angular.js based Docker web UI\nCode: https://github.com/crosbymichael/dockerui\nDocker Libraries # If you want to write some code to interact with Docker, there is most likely already a binding for your programming language. Check out the link in the documentation to find what is available. If there isn\u0026rsquo;t one available for your language of choice, feel free to create your own, and let us know so we can update the documentation.\nDocker Library list in the Docker Docs\nWhat can I do to help? # If you are a big fan of Docker, and want to know how to help out, then look at the list below, and see if any of them are things that you can do.\nContribute to Docker, it could be as small as a bug fix, documentation update, or a new feature. Look through the docker issues, and see if anything tickles your fancy. Tweet about how much you love Docker Write a blog post about how you use Docker, and how others can do what you have done. Talk at a conference or meetup. This is a good way to introduce docker to a new set of potential Docker lovers. Create a product that uses Docker, and let everyone know how Docker made your life easier. Make a video showing how you use Docker, and upload to YouTube/Vimeo. Answer questions on : - Stack Overflow - IRC - Mailing list Attend the Docker hack days and meet other Docker users, and let us know how we can make Docker even better. Get a Docker sticker, and display it proudly. Wear your Docker shirt around town all day. Tips and Tricks # This section includes some helpful tips and tricks that will make using Docker even more easier and fun.\nRemove all Docker images # $ docker rmi `docker images -a -q` Remove all Docker containers # $ docker rm `docker ps -a -q` Docker Commands # Here is a list of all of the current Docker commands, the different parameters they might have, as well as an example or two on how to use them.\nattach # Attach to a running container. To disconnect press Ctrl+P, Ctrl+Q.\nParameters # CONTAINER_ID: The ID for the container you want to attach too. Usage # docker attach CONTAINER_ID Example # docker attach afs232ybh2123d # To disconnect press Ctrl+P, Ctrl+Q. build # Build a container from a Dockerfile\nParameters # PATH: Build a new container image from the source code at PATH URL: When a single Dockerfile is given as URL, then no context is set. When a git repository is set as URL, the repository is used as context OPTIONS: : - -t=\u0026quot;\u0026quot; : Tag to be applied to the resulting image in case of success. Usage # docker build [OPTIONS] PATH | URL | - Examples # Read the Dockerfile from the current directory # docker build . This will read the Dockerfile from the current directory. It will also send any other files and directories found in the current directory to the docker daemon. The contents of this directory would be used by ADD commands found within the Dockerfile. This will send a lot of data to the docker daemon if the current directory contains a lot of data. If the absolute path is provided instead of ‘.’, only the files and directories required by the ADD commands from the Dockerfile will be added to the context and transferred to the docker daemon.\nRead a Dockerfile from standard in (stdin) without context # docker build - \u0026lt; Dockerfile This will read a Dockerfile from Stdin without context. Due to the lack of a context, no contents of any local directory will be sent to the docker daemon. ADD doesn’t work when running in this mode due to the absence of the context, thus having no source files to copy to the container.\nBuild from a git repo # docker build github.com/creack/docker-firefox This will clone the github repository and use it as context. The Dockerfile at the root of the repository is used as Dockerfile. Note that you can specify an arbitrary git repository by using the ‘git://’ schema.\ncommit # Save your containers state to a container image, so the state can be re-used.\nWhen you commit your container only the differences between the image the container was created from and the current state of the container will be stored (as a diff). See which images you already have using docker images\nIn order to commit to the repository it is required to have committed your container to an image with your namespace.\nParameters # CONTAINER_ID: The container ID for the container you want to commit REPOSITORY: The name for your image that you will save to the repository \u0026lt;your username\u0026gt;/\u0026lt;image name\u0026gt; TAG: The tag you want to give to the commit. OPTIONS: : - -m=\u0026quot;\u0026quot;: Commit message - -author=\u0026quot;\u0026quot;: Author (eg. \u0026quot;John Hannibal Smith \u0026amp;lt;\u0026lt;hannibal@a-team.com\u0026gt;\u0026amp;gt;\u0026quot; - -run=\u0026quot;\u0026quot;: Config automatically applied when the image is run. \u0026quot;+\\`(ex: {\u0026quot;Cmd\u0026quot;: \\[\u0026quot;cat\u0026quot;, \u0026quot;/world\u0026quot;\\], \u0026quot;PortSpecs\u0026quot;: \\[\u0026quot;22\u0026quot;\\]}') Usage # docker commit [OPTIONS] CONTAINER_ID [REPOSITORY [TAG]] Examples # basic commit # This will commit a container with a message and author. :\ndocker commit -m=\u0026quot;My commit message\u0026quot; -author=\u0026quot;Joe smith\u0026quot; a1bcbabsdhb323h2b commit with repository # Same as basic commit, but with a repository name :\ndocker commit -m=\u0026quot;My commit message\u0026quot; -author=\u0026quot;Joe smith\u0026quot; a1bcbabsdhb323h2b joesmith/myrepo commit with tag # Same as basic commit, but with a repository name and tag :\ndocker commit -m=\u0026quot;My commit message\u0026quot; -author=\u0026quot;Joe smith\u0026quot; a1bcbabsdhb323h2b joesmith/myrepo mytag Full example # An example with all parameters and options. :\ndocker commit -m=\u0026quot;My commit message\u0026quot; -author=\u0026quot;Joe smith\u0026quot; -run='{\u0026quot;Hostname\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;,\u0026quot;CpuShares\u0026quot;: 0,\u0026quot;Memory\u0026quot;: 0,\u0026quot;MemorySwap\u0026quot;: 0,\u0026quot;PortSpecs\u0026quot;: [\u0026quot;22\u0026quot;, \u0026quot;80\u0026quot;, \u0026quot;443\u0026quot;],\u0026quot;Tty\u0026quot;: true,\u0026quot;OpenStdin\u0026quot;: true,\u0026quot;StdinOnce\u0026quot;: true,\u0026quot;Env\u0026quot;: [\u0026quot;FOO=BAR\u0026quot;, \u0026quot;FOO2=BAR2\u0026quot;],\u0026quot;Cmd\u0026quot;: [\u0026quot;cat\u0026quot;, \u0026quot;-e\u0026quot;, \u0026quot;/etc/resolv.conf\u0026quot;],\u0026quot;Dns\u0026quot;: [\u0026quot;8.8.8.8\u0026quot;, \u0026quot;8.8.4.4\u0026quot;]}' a1bcbabsdhb323h2b joesmith/myrepo mytag diff # Inspect changes on a container’s filesystem\nParameters # CONTAINER_ID: The ID for the container you want to create a diff for Usage # docker diff CONTAINER_ID Examples # docker diff a1bcbabsdhb323h2b export # Stream the contents of a container as a tar archive\nParameters # CONTAINER_ID: The ID for the container you want to export. Usage # docker export CONTAINER_ID Examples # docker export a1bcbabsdhb323h2b \u0026gt; myfile.tar history # Show the history of an image\nParameters # IMAGE: The name of the image you want to see the history for Usage # docker history IMAGE Examples # docker history joesmith/myimage images # List the images managed by Docker\nParameters # NAME: A filter to limit results to only images matching the NAME OPTIONS: : - -a=false: show all images - -q=false: only show numeric IDs - -viz=false: output in graphviz format Usage # docker images [OPTIONS] [NAME] Examples # Show images # docker images Show images with name ubuntu # docker images ubuntu Show all images # docker images -a Show only image ID\u0026rsquo;s # docker images -q Displaying images visually # docker images -viz | dot -Tpng -o docker.png import # Create a new filesystem image from the contents of a tarball\nParameters # URL: At this time, the URL must start with http and point to a single file archive (.tar, .tar.gz, .bzip) containing a root filesystem. If you would like to import from a local directory or archive, you can use the - parameter to take the data from standard in. TAG: name of the tag you want to assign repo after import REPOSITORY: the repository to import into. Usage # docker import URL |- [REPOSITORY [TAG]] Examples # Import from a remote location # $ docker import http://example.com/exampleimage.tgz exampleimagerepo Import from a local file # Import to docker via pipe and standard in:\n$ cat exampleimage.tgz | docker import - exampleimagelocal Import from a local directory # Note the sudo in this example – you must preserve the ownership of the files (especially root ownership) during the archiving with tar. If you are not root (or sudo) when you tar, then the ownerships might not get preserved. :\n$ sudo tar -c . | docker import - exampleimagedir info # Display system-wide information.\nParameters # None\nUsage # $ docker info Examples # $ docker info Containers: 30 Images: 25 Debug mode (server): true Debug mode (client): false Fds: 8 Goroutines: 10 inspect # Return low-level information on a container/image. The command will take 1 or more container or image ids and return all of the information relating to those ids.\nParameters # CONTAINER: The ID for the container you want to export. IMAGE: The image name for the images you want information for. Usage # $ docker inspect CONTAINER|IMAGE [CONTAINER|IMAGE...] Examples # Container inspect # Inspect one container :\n$ docker inspect a5e78640ece4 [{ \u0026quot;ID\u0026quot;: \u0026quot;a5e78640ece4b64657b86780ebfeacf614c402cf3b30bb2226f9f8abd48a46ff\u0026quot;, \u0026quot;Created\u0026quot;: \u0026quot;2013-07-05T22:43:36.281232878Z\u0026quot;, \u0026quot;Path\u0026quot;: \u0026quot;sh\u0026quot;, \u0026quot;Args\u0026quot;: [], \u0026quot;Config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;a5e78640ece4\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: true, \u0026quot;AttachStdout\u0026quot;: true, \u0026quot;AttachStderr\u0026quot;: true, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: true, \u0026quot;OpenStdin\u0026quot;: true, \u0026quot;StdinOnce\u0026quot;: true, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: [ \u0026quot;sh\u0026quot; ], \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;joffrey/busybox\u0026quot;, \u0026quot;Volumes\u0026quot;: {}, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: [] }, \u0026quot;State\u0026quot;: { \u0026quot;Running\u0026quot;: false, \u0026quot;Pid\u0026quot;: 0, \u0026quot;ExitCode\u0026quot;: 0, \u0026quot;StartedAt\u0026quot;: \u0026quot;2013-07-05T22:43:36.286163881Z\u0026quot;, \u0026quot;Ghost\u0026quot;: false }, \u0026quot;Image\u0026quot;: \u0026quot;e74096c5172b34732c9769db5f23805cf786dffe25f25da66ebf7c0fc30d0e0b\u0026quot;, \u0026quot;NetworkSettings\u0026quot;: { \u0026quot;IPAddress\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;IPPrefixLen\u0026quot;: 0, \u0026quot;Gateway\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Bridge\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;PortMapping\u0026quot;: null }, \u0026quot;SysInitPath\u0026quot;: \u0026quot;/usr/bin/docker\u0026quot;, \u0026quot;ResolvConfPath\u0026quot;: \u0026quot;/etc/resolv.conf\u0026quot;, \u0026quot;Volumes\u0026quot;: {}, \u0026quot;VolumesRW\u0026quot;: {} }] Inspect more then one container # Inspect 2 containers :\n$ docker inspect a5e78640ece4 0775b219a48a [{ \u0026quot;ID\u0026quot;: \u0026quot;a5e78640ece4b64657b86780ebfeacf614c402cf3b30bb2226f9f8abd48a46ff\u0026quot;, \u0026quot;Created\u0026quot;: \u0026quot;2013-07-05T22:43:36.281232878Z\u0026quot;, \u0026quot;Path\u0026quot;: \u0026quot;sh\u0026quot;, \u0026quot;Args\u0026quot;: [], \u0026quot;Config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;a5e78640ece4\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: true, \u0026quot;AttachStdout\u0026quot;: true, \u0026quot;AttachStderr\u0026quot;: true, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: true, \u0026quot;OpenStdin\u0026quot;: true, \u0026quot;StdinOnce\u0026quot;: true, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: [ \u0026quot;sh\u0026quot; ], \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;joffrey/busybox\u0026quot;, \u0026quot;Volumes\u0026quot;: {}, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: [] }, \u0026quot;State\u0026quot;: { \u0026quot;Running\u0026quot;: false, \u0026quot;Pid\u0026quot;: 0, \u0026quot;ExitCode\u0026quot;: 0, \u0026quot;StartedAt\u0026quot;: \u0026quot;2013-07-05T22:43:36.286163881Z\u0026quot;, \u0026quot;Ghost\u0026quot;: false }, \u0026quot;Image\u0026quot;: \u0026quot;e74096c5172b34732c9769db5f23805cf786dffe25f25da66ebf7c0fc30d0e0b\u0026quot;, \u0026quot;NetworkSettings\u0026quot;: { \u0026quot;IPAddress\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;IPPrefixLen\u0026quot;: 0, \u0026quot;Gateway\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Bridge\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;PortMapping\u0026quot;: null }, \u0026quot;SysInitPath\u0026quot;: \u0026quot;/usr/bin/docker\u0026quot;, \u0026quot;ResolvConfPath\u0026quot;: \u0026quot;/etc/resolv.conf\u0026quot;, \u0026quot;Volumes\u0026quot;: {}, \u0026quot;VolumesRW\u0026quot;: {} },{ \u0026quot;ID\u0026quot;: \u0026quot;0775b219a48ab9bbebe841a0388f9909e996140f941585e318dbe64289392534\u0026quot;, \u0026quot;Created\u0026quot;: \u0026quot;2013-07-05T22:40:47.219244957Z\u0026quot;, \u0026quot;Path\u0026quot;: \u0026quot;sh\u0026quot;, \u0026quot;Args\u0026quot;: [], \u0026quot;Config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;0775b219a48a\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: true, \u0026quot;AttachStdout\u0026quot;: true, \u0026quot;AttachStderr\u0026quot;: true, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: true, \u0026quot;OpenStdin\u0026quot;: true, \u0026quot;StdinOnce\u0026quot;: true, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: [ \u0026quot;sh\u0026quot; ], \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;joffrey/busybox\u0026quot;, \u0026quot;Volumes\u0026quot;: {}, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: [] }, \u0026quot;State\u0026quot;: { \u0026quot;Running\u0026quot;: false, \u0026quot;Pid\u0026quot;: 0, \u0026quot;ExitCode\u0026quot;: 127, \u0026quot;StartedAt\u0026quot;: \u0026quot;2013-07-05T22:40:47.224570459Z\u0026quot;, \u0026quot;Ghost\u0026quot;: false }, \u0026quot;Image\u0026quot;: \u0026quot;e74096c5172b34732c9769db5f23805cf786dffe25f25da66ebf7c0fc30d0e0b\u0026quot;, \u0026quot;NetworkSettings\u0026quot;: { \u0026quot;IPAddress\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;IPPrefixLen\u0026quot;: 0, \u0026quot;Gateway\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Bridge\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;PortMapping\u0026quot;: null }, \u0026quot;SysInitPath\u0026quot;: \u0026quot;/usr/bin/docker\u0026quot;, \u0026quot;ResolvConfPath\u0026quot;: \u0026quot;/etc/resolv.conf\u0026quot;, \u0026quot;Volumes\u0026quot;: {}, \u0026quot;VolumesRW\u0026quot;: {} }] Image inspect # Inspect an Image:\n$ docker inspect bced7ad27b98 [{ \u0026quot;id\u0026quot;: \u0026quot;bced7ad27b98ea990fae3a7479632419109c7a14412365af379a26393ca0492b\u0026quot;, \u0026quot;parent\u0026quot;: \u0026quot;c7fe644d47bc05b6990fafec2f4b61fa0c9f7b248af6e754cbcd9c9507af36b1\u0026quot;, \u0026quot;created\u0026quot;: \u0026quot;2013-06-28T16:45:01.056208611Z\u0026quot;, \u0026quot;container\u0026quot;: \u0026quot;2deff3a37f8b5e1ce6e23ce420be07609df3813429909e2cfe5426c46f0a9552\u0026quot;, \u0026quot;container_config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;2deff3a37f8b\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: false, \u0026quot;AttachStdout\u0026quot;: false, \u0026quot;AttachStderr\u0026quot;: false, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: false, \u0026quot;OpenStdin\u0026quot;: false, \u0026quot;StdinOnce\u0026quot;: false, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: [ \u0026quot;/bin/sh\u0026quot;, \u0026quot;-c\u0026quot;, \u0026quot;apt-get install -y curl\u0026quot; ], \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;c7fe644d47bc\u0026quot;, \u0026quot;Volumes\u0026quot;: null, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: null }, \u0026quot;docker_version\u0026quot;: \u0026quot;0.4.6\u0026quot;, \u0026quot;author\u0026quot;: \u0026quot;Ken \\\u0026quot;ken@example.com\\\u0026quot;\u0026quot;, \u0026quot;config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: false, \u0026quot;AttachStdout\u0026quot;: false, \u0026quot;AttachStderr\u0026quot;: false, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: false, \u0026quot;OpenStdin\u0026quot;: false, \u0026quot;StdinOnce\u0026quot;: false, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: null, \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Volumes\u0026quot;: null, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: null }, \u0026quot;architecture\u0026quot;: \u0026quot;x86_64\u0026quot;, \u0026quot;Size\u0026quot;: 4096 }] Multiple Image inspect # Inspect more then one image at a time:\n$ docker inspect bced7ad27b98 e74096c5172b [{ \u0026quot;id\u0026quot;: \u0026quot;bced7ad27b98ea990fae3a7479632419109c7a14412365af379a26393ca0492b\u0026quot;, \u0026quot;parent\u0026quot;: \u0026quot;c7fe644d47bc05b6990fafec2f4b61fa0c9f7b248af6e754cbcd9c9507af36b1\u0026quot;, \u0026quot;created\u0026quot;: \u0026quot;2013-06-28T16:45:01.056208611Z\u0026quot;, \u0026quot;container\u0026quot;: \u0026quot;2deff3a37f8b5e1ce6e23ce420be07609df3813429909e2cfe5426c46f0a9552\u0026quot;, \u0026quot;container_config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;2deff3a37f8b\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: false, \u0026quot;AttachStdout\u0026quot;: false, \u0026quot;AttachStderr\u0026quot;: false, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: false, \u0026quot;OpenStdin\u0026quot;: false, \u0026quot;StdinOnce\u0026quot;: false, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: [ \u0026quot;/bin/sh\u0026quot;, \u0026quot;-c\u0026quot;, \u0026quot;apt-get install -y curl\u0026quot; ], \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;c7fe644d47bc\u0026quot;, \u0026quot;Volumes\u0026quot;: null, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: null }, \u0026quot;docker_version\u0026quot;: \u0026quot;0.4.6\u0026quot;, \u0026quot;author\u0026quot;: \u0026quot;Ken \\\u0026quot;ken@example.com\\\u0026quot;\u0026quot;, \u0026quot;config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: false, \u0026quot;AttachStdout\u0026quot;: false, \u0026quot;AttachStderr\u0026quot;: false, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: false, \u0026quot;OpenStdin\u0026quot;: false, \u0026quot;StdinOnce\u0026quot;: false, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: null, \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Volumes\u0026quot;: null, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: null }, \u0026quot;architecture\u0026quot;: \u0026quot;x86_64\u0026quot;, \u0026quot;Size\u0026quot;: 4096 },{ \u0026quot;id\u0026quot;: \u0026quot;e74096c5172b34732c9769db5f23805cf786dffe25f25da66ebf7c0fc30d0e0b\u0026quot;, \u0026quot;parent\u0026quot;: \u0026quot;e9aa60c60128cad1\u0026quot;, \u0026quot;created\u0026quot;: \u0026quot;2013-05-09T09:45:26.287021-07:00\u0026quot;, \u0026quot;container\u0026quot;: \u0026quot;73f9f76d46cc07b3a6aa4e96c85dbabbfc4d1345697f263d5cd1741b5b05d6f2\u0026quot;, \u0026quot;container_config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;73f9f76d46cc\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: false, \u0026quot;AttachStdout\u0026quot;: true, \u0026quot;AttachStderr\u0026quot;: true, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: false, \u0026quot;OpenStdin\u0026quot;: false, \u0026quot;StdinOnce\u0026quot;: false, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: [ \u0026quot;ls\u0026quot; ], \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;busybox\u0026quot;, \u0026quot;Volumes\u0026quot;: {}, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: null }, \u0026quot;docker_version\u0026quot;: \u0026quot;0.3.0\u0026quot;, \u0026quot;Size\u0026quot;: 16391 }] Container and Image inspect # Inspect a container and an image at the same time:\n$ docker inspect bced7ad27b98 a5e78640ece4 [{ \u0026quot;id\u0026quot;: \u0026quot;bced7ad27b98ea990fae3a7479632419109c7a14412365af379a26393ca0492b\u0026quot;, \u0026quot;parent\u0026quot;: \u0026quot;c7fe644d47bc05b6990fafec2f4b61fa0c9f7b248af6e754cbcd9c9507af36b1\u0026quot;, \u0026quot;created\u0026quot;: \u0026quot;2013-06-28T16:45:01.056208611Z\u0026quot;, \u0026quot;container\u0026quot;: \u0026quot;2deff3a37f8b5e1ce6e23ce420be07609df3813429909e2cfe5426c46f0a9552\u0026quot;, \u0026quot;container_config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;2deff3a37f8b\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: false, \u0026quot;AttachStdout\u0026quot;: false, \u0026quot;AttachStderr\u0026quot;: false, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: false, \u0026quot;OpenStdin\u0026quot;: false, \u0026quot;StdinOnce\u0026quot;: false, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: [ \u0026quot;/bin/sh\u0026quot;, \u0026quot;-c\u0026quot;, \u0026quot;apt-get install -y curl\u0026quot; ], \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;c7fe644d47bc\u0026quot;, \u0026quot;Volumes\u0026quot;: null, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: null }, \u0026quot;docker_version\u0026quot;: \u0026quot;0.4.6\u0026quot;, \u0026quot;author\u0026quot;: \u0026quot;Ken \\\u0026quot;ken@dotcloud.com\\\u0026quot;\u0026quot;, \u0026quot;config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: false, \u0026quot;AttachStdout\u0026quot;: false, \u0026quot;AttachStderr\u0026quot;: false, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: false, \u0026quot;OpenStdin\u0026quot;: false, \u0026quot;StdinOnce\u0026quot;: false, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: null, \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Volumes\u0026quot;: null, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: null }, \u0026quot;architecture\u0026quot;: \u0026quot;x86_64\u0026quot;, \u0026quot;Size\u0026quot;: 4096 },{ \u0026quot;ID\u0026quot;: \u0026quot;a5e78640ece4b64657b86780ebfeacf614c402cf3b30bb2226f9f8abd48a46ff\u0026quot;, \u0026quot;Created\u0026quot;: \u0026quot;2013-07-05T22:43:36.281232878Z\u0026quot;, \u0026quot;Path\u0026quot;: \u0026quot;sh\u0026quot;, \u0026quot;Args\u0026quot;: [], \u0026quot;Config\u0026quot;: { \u0026quot;Hostname\u0026quot;: \u0026quot;a5e78640ece4\u0026quot;, \u0026quot;User\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Memory\u0026quot;: 0, \u0026quot;MemorySwap\u0026quot;: 0, \u0026quot;CpuShares\u0026quot;: 0, \u0026quot;AttachStdin\u0026quot;: true, \u0026quot;AttachStdout\u0026quot;: true, \u0026quot;AttachStderr\u0026quot;: true, \u0026quot;PortSpecs\u0026quot;: null, \u0026quot;Tty\u0026quot;: true, \u0026quot;OpenStdin\u0026quot;: true, \u0026quot;StdinOnce\u0026quot;: true, \u0026quot;Env\u0026quot;: null, \u0026quot;Cmd\u0026quot;: [ \u0026quot;sh\u0026quot; ], \u0026quot;Dns\u0026quot;: null, \u0026quot;Image\u0026quot;: \u0026quot;joffrey/busybox\u0026quot;, \u0026quot;Volumes\u0026quot;: {}, \u0026quot;VolumesFrom\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Entrypoint\u0026quot;: [] }, \u0026quot;State\u0026quot;: { \u0026quot;Running\u0026quot;: false, \u0026quot;Pid\u0026quot;: 0, \u0026quot;ExitCode\u0026quot;: 0, \u0026quot;StartedAt\u0026quot;: \u0026quot;2013-07-05T22:43:36.286163881Z\u0026quot;, \u0026quot;Ghost\u0026quot;: false }, \u0026quot;Image\u0026quot;: \u0026quot;e74096c5172b34732c9769db5f23805cf786dffe25f25da66ebf7c0fc30d0e0b\u0026quot;, \u0026quot;NetworkSettings\u0026quot;: { \u0026quot;IPAddress\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;IPPrefixLen\u0026quot;: 0, \u0026quot;Gateway\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;Bridge\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;PortMapping\u0026quot;: null }, \u0026quot;SysInitPath\u0026quot;: \u0026quot;/usr/bin/docker\u0026quot;, \u0026quot;ResolvConfPath\u0026quot;: \u0026quot;/etc/resolv.conf\u0026quot;, \u0026quot;Volumes\u0026quot;: {}, \u0026quot;VolumesRW\u0026quot;: {} }] kill # Kill a running container(s). If the container won\u0026rsquo;t stop, you can brute force it with the kill command.\nParameters # CONTAINER: The container id for the container you want to kill, can be one or a list separated by spaces. Usage # $ docker kill CONTAINER [CONTAINER...] Examples # Kill one container # $ docker kill a5e78640ece4 a5e78640ece4 Kill more then one container # $ docker kill a5e78640ece4 0775b219a48a a5e78640ece4 0775b219a48a login # Register or Login to the docker registry server. If you have an account it will log you in, and cache the credentials, if you don\u0026rsquo;t have an account it will create one for you, and automatically log you in. You can pass in the username, email and password as command line parameters to easily script out the login process.\nParameters # OPTIONS: : - e: email - p: password - u: username Usage # $ docker login [OPTIONS] Examples # Login with prompts # $ docker login Username (): myusername Password: Email (): myusername@example.com Login Succeeded Login with parameters # $ docker login -u myusername -p mypassword -e myusername@example.com Login Succeeded logs # Fetch the logs of a container\nParameters # CONTAINER: The Container ID for the Container you want to get the logs for. Usage # $ docker logs CONTAINER Examples # $ docker logs a5e78640ece4 some logs from my container some logs from my container some logs from my container ... port # Lookup the public-facing port which is NAT-ed to PRIVATE_PORT\nParameters # CONTAINER: The Container ID for the container you want to find the port for PRIVATE_PORT: The private port, you want to find the matching Public port for Usage # $ docker port CONTAINER PRIVATE_PORT Examples # $ docker port 335c587d6ad1 6379 49153 ps # List containers\nParameters # OPTIONS: : - -a=false: Show all containers. Only running containers are shown by default. - -notrunc=false: Don't truncate output - -q=false: Only display numeric IDs Usage # docker ps [OPTIONS] Examples # Show running containers # $ docker ps ID IMAGE COMMAND CREATED STATUS PORTS 335c587d6ad1 joffrey/busybox:latest /bin/sh -c while tru 3 minutes ago Up 3 minutes 49153-\u0026gt;6379 Show all containers # $ docker ps -a ID IMAGE COMMAND CREATED STATUS PORTS 335c587d6ad1 joffrey/busybox:latest /bin/sh -c while tru 3 minutes ago Up 3 minutes 49153-\u0026gt;6379 1347dbb9d32f joffrey/busybox:latest /bin/sh -c while tru 4 minutes ago Exit 137 db2db67170ba joffrey/busybox:latest /bin/echo hi 5 minutes ago Exit 0 a5e78640ece4 joffrey/busybox:latest sh 6 days ago Exit 0 0775b219a48a joffrey/busybox:latest sh 6 days ago Exit 127 1668f16b3ef4 joffrey/busybox:latest bash 6 days ago Exit 127 ... trimed show all containers full output # $ docker ps -a -notrunc ID IMAGE COMMAND CREATED STATUS PORTS 335c587d6ad121519e1489b837e80a5efb748669c86a8bdd485867759fb3c9a7 joffrey/busybox:latest /bin/sh -c while true; do echo hello world; sleep 1; done 4 minutes ago Up 4 minutes 49153-\u0026gt;6379 1347dbb9d32fcafe922a58e6b01c56d04d35fbd3f3226e3789c30310222eceee joffrey/busybox:latest /bin/sh -c while true; do echo hello world; sleep 1; done 5 minutes ago Exit 137 db2db67170ba9e1df14cadcaa6f172ad743b387eea3a9c454001279649463cdb joffrey/busybox:latest /bin/echo hi 6 minutes ago Exit 0 ... Trimmed show only container ids # $ docker ps -q -a 335c587d6ad1 1347dbb9d32f db2db67170ba a5e78640ece4 0775b219a48a ... trimmed pull # Pull an image or a repository from the docker registry server. By default it will always pull down the latest version, but you can also pull by tag.\nParameters # NAME: the name of the repository to pull from registry OPTIONS: : - -t: Tag, if you want to pull down a tagged version of the repository. Usage # $ docker pull NAME Examples # Pull library repository # $ docker pull base Pull User repository # $ docker pull samalba/hipache Pull repository by tag # replace latest with the tag name you want to pull. :\n$ docker pull samalba/hipache:latest or use the command line flag -t\n$ docker pull -t latest samalba/hipache push # Push an image or a repository to the docker registry server\nParameters # NAME: the name of the repository to push to the registry Usage # $ docker push NAME Examples # $ docker push kencochrane/testrepo restart # Restart one or more running containers\nParameters # CONTAINER: The Container ID for the container you want to restart OPTIONS: : - t: Number of seconds to try to stop for before killing the container. Once killed it will then be restarted Usage # $ docker restart [OPTIONS] CONTAINER [CONTAINER ...] Examples # restart container # $ docker restart 335c587d6ad1 335c587d6ad1 restart multiple containers # $ docker restart 335c587d6ad1 1347dbb9d32f 335c587d6ad1 1347dbb9d32f restart container with 15 second timeout # $ docker restart -t 15 335c587d6ad1 335c587d6ad1 rm # Remove a container\nParameters # CONTAINER: The Container ID for the container you want to remove OPTIONS: : - v: Remove the volumes associated to the container Usage # $ docker rm [OPTIONS] CONTAINER Examples # Remove container # $ docker rm 335c587d6ad1 Remove container and volume # $ docker rm -v 335c587d6ad1 rmi # Remove one or more images\nParameters # IMAGE: The ID for the image you want to remove Usage # $ docker rmi IMAGE [IMAGE...] Examples # Remove one image # $ docker rmi bced7ad27b98 Remove more then one image # $ docker rmi bced7ad27b98 e74096c5172b run # Run a command in a new container\nParameters # IMAGE: The name of the image you want to create a container from OPTIONS:\na=map[]: Attach to stdin, stdout or stderr. c=0: CPU shares (relative weight) d=false: Detached mode: leave the container running in the background e=[]: Set environment variables h=\u0026quot;\u0026quot;: Container host name i=false: Keep stdin open even if not attached m=0: Memory limit (in bytes) p=[]: Map a network port to the container t=false: Allocate a pseudo-tty u=\u0026quot;\u0026quot;: Username or UID d=[]: Set custom dns servers for the container v=[]: Creates a new volume and mounts it at the specified path. volumes-from=\u0026quot;\u0026quot;: Mount all volumes from the given container. b=[]: Create a bind mount with: [host-dir]:[container-dir]:[rw|ro] entrypoint=\u0026quot;\u0026quot;: Overwrite the default entrypoint set by the image. Usage # $ docker run [OPTIONS] IMAGE [COMMAND] [ARG...] Examples # Run container in foreground # TODO:\nRun container in background # TODO:\nStart container with memory limit # TODO:\nLimit containers CPU shares # TODO:\nSet container environment variables # TODO:\nAttach a Volume to a container # TODO:\nSet custom DBS server for the container # TODO:\nCreate bind mount for container # TODO:\nOverride the default entrypoint set by image # TODO:\nsearch # Search for an image in the docker index\nParameters # TERM: Search term OPTIONS: : - notrunc Usage # $ docker search [OPTIONS] TERM Examples # Normal search # $ docker search base Show full results # This will not truncate the description field for the search results :\n$ docker search -notrunc base start # Start one or more stopped containers\nParameters # CONTAINER: The container ID for the container you want to start Usage # $ docker start CONTAINER [CONTAINER...] Examples # Start one container # $ docker start 335c587d6ad1 335c587d6ad1 Start two containers # $ docker start 335c587d6ad1 1347dbb9d32f 335c587d6ad1 1347dbb9d32f stop # Stop a running container\nParameters # CONTAINER: The container ID for the container you want to stop OPTIONS: : - t=10: Number of seconds to try to stop for before killing the container. Usage # $ docker stop [OPTIONS] CONTAINER [CONTAINER...] Examples # Stop one container # $ docker stop 335c587d6ad1 335c587d6ad1 Stop two containers # $ docker stop 335c587d6ad1 1347dbb9d32f 335c587d6ad1 1347dbb9d32f Stop container with 15 second timeout # $ docker stop -t 15 335c587d6ad1 335c587d6ad1 tag # Tag an image into a repository\nParameters # IMAGE: The image to tag REPOSITORY: The repository name in the registry TAG: The tag name OPTIONS: : - f=false: Force Usage # $ docker tag [OPTIONS] IMAGE REPOSITORY [TAG] Examples # Tag an image # TODO:\nTag an image, without specifying a Tag # TODO:\nForce setting a Tag # TODO:\nversion # Show the docker version information\nParameters # None\nUsage # $ docker version Examples # $ docker version Client version: 0.5.0 Server version: 0.5.0 Go version: go1.1 wait # Block until a container stops, then print its exit code\nParameters # CONTAINER: The container ID for the container you want to wait for Usage # $ docker wait CONTAINER Examples # $ docker wait 335c587d6ad1 0 ","date":"August 23, 2013","externalUrl":null,"permalink":"/blog/2013/08/the-docker-guidebook/","section":"Posts","summary":"","title":"The Docker Guidebook","type":"post"},{"content":"","date":"August 23, 2013","externalUrl":null,"permalink":"/tags/tutorial/","section":"Tags","summary":"","title":"Tutorial","type":"tags"},{"content":"","date":"June 7, 2013","externalUrl":null,"permalink":"/tags/aufs/","section":"Tags","summary":"","title":"Aufs","type":"tags"},{"content":"","date":"June 7, 2013","externalUrl":null,"permalink":"/tags/digital-ocean/","section":"Tags","summary":"","title":"Digital-Ocean","type":"tags"},{"content":"","date":"June 7, 2013","externalUrl":null,"permalink":"/tags/kernel/","section":"Tags","summary":"","title":"Kernel","type":"tags"},{"content":"","date":"June 7, 2013","externalUrl":null,"permalink":"/tags/lxc/","section":"Tags","summary":"","title":"Lxc","type":"tags"},{"content":"I recently wrote a post on how to get Docker up and running on Rackspace and since then I have received some requests on how to get it up and running on other hosts. One of those hosts is Digital Ocean a hot new Cloud hosting provider that offers a 512MB 20GB SSD VPS for only $5.00 / month. A really great deal, and also a great price if you want to play around with some new tool and not have to worry about possibly breaking a production server. This post should guide you though the process of getting an Ubuntu 12.04 and 13.04 VPS up and running with Docker.\nStep 1: Register # First things first, if you don\u0026rsquo;t already have a Digital Ocean account, you will need to create one. If you follow this link, click sign up and enter this promo code VPSERS10, you will be given a $10 credit to try out the service.\nStep 2: Billing # To prevent abuse, Digital Ocean requires that you enter a credit card before you can spin up a server. Go ahead and do that now. Once you put in your information you should see a screen like this.\n{.img-polaroid}\nStep 3 SSH keys # To make your life easier, I would add a public SSH key to your account. If you don\u0026rsquo;t add an SSH key then you will be emailed a root password when the server is setup. It is easier and more secure to add your public key, and then select that key when building the server, and they will automatically add the key to your server for you. If you don\u0026rsquo;t have a SSH key, don\u0026rsquo;t worry they are easy to create. Just do a quick web search, and there are a ton of different guide out there to help you get one setup.\n{.img-polaroid}\nStep 4: Create Droplet # Digital Ocean calls their servers Droplets. Lets create a droplet. Click on the big \u0026ldquo;Create\u0026rdquo; button on the control panel.\nEnter a hostname at the top. Pick your size select your region Select your image. Docker currently only runs on a 64bit OS, and needs a fairly recent kernel (3.8+) with AUFS enabled. The only images on Digital Ocean that will currently work with Docker are Ubuntu 13.04 x64 Server, and Ubuntu 12.04 x64 Server. I\u0026rsquo;ll cover the install instructions for those two options below. Pick your SSH key that you added previously. If you don\u0026rsquo;t pick one, your root password will be emailed to you. Enable VirtIO Click the big \u0026ldquo;Create Droplet\u0026rdquo; button at the bottom. {.img-polaroid}\nDroplet getting created\n{.img-polaroid}\nAfter about 60 seconds you should have a cloudlet created with an IP address. Now pick the distribution you picked below and follow the rest of the directions.\nUbuntu 12.04 64bit Server # Upgrade kernel # The default kernel with 12.04 doesn\u0026rsquo;t work well with Docker so we are going to upgrade to the same one that is used by 13.04. To do this you will go into your control panel for your droplet, and go into the settings tab and change the kernel pull down to \u0026ldquo;Ubuntu 13.04-x64-vmlinuz-3.8.0-23-generic\u0026rdquo; and click change.\n{.img-polaroid}\nIn order for the kernel change to take affect you will need to power cycle the droplet. Click on the power tab, and then hit the \u0026ldquo;Power Cycle\u0026rdquo; button.\n{.img-polaroid}\nLogin to server # Now that you have the new kernel you need to login to the server to install the rest of the stuff.\n$ ssh root@\u0026lt;your_ip_address\u0026gt; Let\u0026rsquo;s check to make sure you have the right kernel. It should show a 3.8 kernel if you did everything right.\n$ uname -a Linux docker-1 3.8.0-23-generic #34-Ubuntu SMP Wed May 29 20:22:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux Let\u0026rsquo;s install some dependencies now.\n$ sudo apt-get update \u0026amp;\u0026amp; sudo apt-get install linux-image-generic-lts-raring If you see this, just pick the default (already selected) and hit OK.\n{.img-polaroid}\nInstall Docker from the Docker PPA\n$ sudo apt-get install python-software-properties \u0026amp;\u0026amp; sudo add-apt-repository ppa:dotcloud/lxc-docker $ sudo apt-get update $ sudo apt-get install lxc-docker Docker should have been installed and started up. lets test to make sure it works.\n$ docker ps ID IMAGE COMMAND CREATED STATUS PORTS} $ docker version Client version: 0.4.0 Server version: 0.4.0 Go version: go1.0.3 $ docker run base /bin/echo hello world hello world Hopefully it worked. If not, feel free to ask questions on #docker on freenode, or submit a support ticket.\nUbuntu 13.04 64bit Server # 13.04 comes with the 3.8 kernel, so we won\u0026rsquo;t need to do anything kernel related, which makes this install much simpler compared to 12.04.\nInstall the dependencies\n$ sudo apt-get update $ sudo apt-get install linux-image-extra-`uname -r` If you see this, just pick the default (already selected) and hit OK.\n{.img-polaroid}\nInstall Docker\n$ sudo apt-get install software-properties-common $ sudo add-apt-repository ppa:dotcloud/lxc-docker $ sudo apt-get update $ sudo apt-get install lxc-docker Docker should have been installed and started up. lets test to make sure it works.\n$ docker ps ID IMAGE COMMAND CREATED STATUS PORTS $ docker version Client version: 0.4.0 Server version: 0.4.0 Go version: go1.0.3 $ docker run base /bin/echo hello world hello world Hopefully it worked. If not, feel free to ask questions on #docker on freenode, or submit a support ticket.\nConclusion # Hopefully now you have the knowledge to go and setup your own Docker server on Digital Ocean. If you have any issues, or questions feel free to submit the questions below or visit #docker on freenode\n","date":"June 7, 2013","externalUrl":null,"permalink":"/blog/2013/06/running-docker-on-digital-ocean/","section":"Posts","summary":"","title":"Running Docker on Digital Ocean with Ubuntu","type":"post"},{"content":"","date":"June 7, 2013","externalUrl":null,"permalink":"/tags/ubuntu/","section":"Tags","summary":"","title":"Ubuntu","type":"tags"},{"content":"This year I attended PyCon US and I was lucky enough to get a FREE RaspberryPi. At the same conference dotCloud (The company I work for), was giving a lightning talk for a project that we have been working on, called Docker. Docker is a tool that allows you to better manage your Linux Containers (LXC). Shortly after I got the RaspberryPi, I started wondering if it would be possible to run Docker on it.\n{.img-polaroid}\nI did some digging and technically speaking the Pi should be able to run Docker, it satisfies all of the requirements.\nLinux LXC AUFS Go Getting it up and running isn\u0026rsquo;t going to be easy. The Linux kernels that come with the different Linux distros for the RaspberryPi, are kind of old, and don\u0026rsquo;t come with AUFS built in. The RaspberryPi also runs on an ARM based chip, and it is only 32bit. Currently Docker only supports 64bit OS\u0026rsquo;s. There are plans to add 32 bit support in the future, but it isn\u0026rsquo;t there yet.\nDoing some research I was able to find blog posts on how to get LXC and AUFS up and running on the RaspberryPi. Using those guides, I was able to make some progress but I\u0026rsquo;m not all of the way there yet. I\u0026rsquo;m hoping to describe my steps here so that others can see what I have done, and if they want, help me get over the hump.\nGoals # My goals for the project is to:\nprovide a prebuilt image that people can download that has everything they needed in order to get started. I also want to provide a prebuilt kernel, people can download and use without having to build their own. Port Docker to 32bit so that it will run on RaspberryPi and provide a Debian package for easy install. Instructions # Here are the steps that I used to make it so that my RaspberryPi could run Docker. These are still a work in progress, so please let me know if you have any issues, or you found a better way to do this.\nInstall Linux OS # Download Raspbian and make an SD card (I used the 2013-02-09-wheezy-raspbian.zip image) Once you have the SD card, put it in the Pi and boot it up. Update and Prepare Raspbian # Switch to Root User on the Pi. These commands must be run as root. You can also use \u0026ldquo;su\u0026rdquo; or \u0026ldquo;sudo\u0026rdquo;, what ever you prefer: $ sudo su root Expand to fill SD card and reboot after entering: $ raspi-config Update Raspbian $ apt-get update $ apt-get dist-upgrade Install git $ sudo apt-get install git-core Update Firmware The clone will take a while. You might consider cloning on a desktop machine to save time. Just transfer the firmware/boot and modules/ directories from your desktop PC to the Pi after the checkout. Be aware that checking out on some OS\u0026rsquo;s that are case insensitive, may result in some files being missing. If you see funky issues, this might be the cause.\n$ cd /opt $ git clone git://github.com/raspberrypi/firmware.git $ cd firmware/boot $ cp * /boot $ cd ../modules $ cp -r * /lib/modules $ reboot Increase the Swap File Size I found that in order to check out the source on the Pi, you’ll need a swap file with the 256MB Pi, otherwise it will run out of RAM during the checkout (with fatal: index-pack failed).\n# use your favorite editor here. $ pico /etc/dphys-swapfile # change to 500 (MB) $ sudo dphys-swapfile setup $ sudo reboot Prepare to Build Kernel We are going to use the 3.6 kernel since it is the lastest stable one. There is an effort to get 3.8 working, it isn\u0026rsquo;t 100% there yet, for more info see.\nThe clone will take a while. Again, you may consider using a desktop PC. Of course, if you do that, you’ll need to issue the “zcat” command from your Pi and copy the resulting “.config” file to the “linux” directory on your desktop PC.\n$ cd /opt $ mkdir raspberrypi $ cd raspberrypi $ git clone git://github.com/raspberrypi/linux.git $ cd linux $ zcat /proc/config.gz \u0026gt; .config Decrease the Swap Space File $ pico /etc/dphys-swapfile # change to 100 (MB) $ sudo dphys-swapfile setup $ sudo reboot Install Packages for Kernel Compilation $ apt-get install ncurses-dev Adding AUFS Patches $ cd /opt/raspberrypi/linux git clone git://aufs.git.sourceforge.net/gitroot/aufs/aufs3-standalone.git cd aufs3-standalone git checkout origin/aufs3.6 cp -rp *.patch ../ cp -rp fs ../ cp -rp Documentation/ ../ cp -rp include/ ../ cd .. patch -p1 \u0026lt; aufs3-base.patch patch -p1 \u0026lt; aufs3-proc_map.patch patch -p1 \u0026lt; aufs3-standalone.patch patch -p1 \u0026lt; aufs3-kbuild.patch If you get this error\nroot@raspberrypi:/opt/raspberrypi/linux# patch -p1 \u0026lt; aufs3-kbuild.patch patching file fs/Kconfig patching file fs/Makefile patching file include/linux/Kbuild Hunk #1 FAILED at 66. 1 out of 1 hunk FAILED -- saving rejects to file include/linux/Kbuild.rej Then you will need to manually update include/linux/Kbuild because the patch failed.\nFirst I reverted change on the file, and manually added. the line (below) to line 66, below audit.h\nheader-y += aufs_type.h here is my git diff:\ndiff --git a/include/linux/Kbuild b/include/linux/Kbuild index fa21760..ee029e3 100644 --- a/include/linux/Kbuild +++ b/include/linux/Kbuild @@ -66,6 +66,7 @@ header-y += atmppp.h header-y += atmsap.h header-y += atmsvc.h header-y += audit.h +header-y += aufs_type.h header-y += auto_fs.h header-y += auto_fs4.h header-y += auxvec.h Configuring Kernel You’ll now need to set some kernel options to support LXC, via the menu config tool.\n$ cd /opt/raspberrypi/linx $ make menuconfig You need to enable these options:\nGeneral -\u0026gt; Control Group Support -\u0026gt; Memory Resource Controller for Control Groups (and its three child options) {.img-polaroid}\n(this has high overhead;only enable if you really need it, or else enable and remember to disable using the Kernel command line option “cgroup_disable=memory”) (image not shown)\nGeneral -\u0026gt; Control Group Support -\u0026gt; cpuset support Device Drivers -\u0026gt; Character Devices -\u0026gt; Support multiple instances of devpts {.img-polaroid}\nDevice Drivers -\u0026gt; Network Device Support -\u0026gt; Virtual ethernet pair device {.img-polaroid}\nFile Systems \u0026ndash;\u0026gt; Miscellaneous filesystems -\u0026gt;select \u0026ldquo;Aufs (Advanced multi layered unification filesystem) support (NEW)\u0026rdquo; (mine was the the very bottom) {.img-polaroid}\nBuild Kernel This could take many hours if you compile on the Pi, there are ways to compile on another machine and transfer the kernel to the Pi when completed. If you are in a hurry, use this approach.\n$ make $ make modules_install $ cd /opt/raspberrypi $ git clone git://github.com/raspberrypi/tools.git $ cd tools/mkimage $ python ./imagetool-uncompressed.py /opt/raspberrypi/linux/arch/arm/boot/Image $ cp /boot/kernel.img /boot/kernel-old.img $ cp kernel.img /boot/ $ reboot Download Latest LXC The LXC tools provided with Raspbian are out-of-date, so let\u0026rsquo;s update to the latest version.\n$ mkdir /opt/lxc $ cd /opt/lxc $ git clone https://github.com/lxc/lxc.git $ apt-get install automake libcap-dev $ cd lxc $ ./autogen.sh \u0026amp;\u0026amp; ./configure \u0026amp;\u0026amp; make \u0026amp;\u0026amp; make install Testing the Install\nCheck LXC is happy with your kernel:\n$ lxc-checkconfig User namespace should be “missing” (it checks for a kernel option that no longer exists) and Cgroup namespace should say “required”.\nInstalling Go If you install Go using the Go package that is available (apt-get install golang). You will get a floating point issue. For more information about the floating point issues see this page. http://www.raspberrypi.org/phpBB3/viewtopic.php?p=129647\nTo get it to work, we will compile Go from source. Feel free to change the location on where we are installing it.\n$ sudo apt-get install -y mercurial gcc libc6-dev $ hg clone -u default https://code.google.com/p/go $HOME/go warning: code.google.com certificate with fingerprint 9f:af:b9:ce:b5:10:97:c0:5d:16:90:11:63:78:fa:2f:37:f4:96:79 not verified (check hostfingerprints or web.cacerts config setting) destination directory: go requesting all changes adding changesets adding manifests adding file changes added 14430 changesets with 52478 changes to 7406 files (+5 heads) updating to branch default 3520 files updated, 0 files merged, 0 files removed, 0 files unresolved $ cd $HOME/go/src $ ./all.bash ALL TESTS PASSED --- Installed Go for linux/arm in /home/dfc/go Installed commands in /home/dfc/go/bin If there was an error relating to out of memory, or you couldn’t configure an appropriate swap device, you can skip the test suite by executing\n$ cd $HOME/go $ ./make.bash as an alternative to ./all.bash.\nThe go command needs to be added to your $PATH, you should also edit your profile script (.bashrc, etc) to include these changes.\n$ export PATH=$PATH:$HOME/go/bin $ go version go version devel +30c566874b83 Wed May 08 16:06:25 2013 -0700 linux/arm Installing Docker $ apt-get -y install wget bsdtar curl git export GOPATH=~/docker/go/ export PATH=$GOPATH/bin:$PATH $ mkdir -p $GOPATH/src/github.com/dotcloud $ cd $GOPATH/src/github.com/dotcloud $ git clone git://github.com/dotcloud/docker.git # or clone your own fork/branch $ cd docker $ go get -v github.com/dotcloud/docker/... $ go install -v github.com/dotcloud/docker/... $ docker version $ docker -d The docker runtime currently only supports amd64 (not arm). This will change in the future. Aborting. Docker is installed but due to current limitations it won\u0026rsquo;t run. It is a start, we now have a development environment to start hacking on Docker to get it to work with the RaspberryPi.\nWhat\u0026rsquo;s Next # Now we have everything up and running, we need to change docker so that it will work on the ARM with only 32bit support. I need to take my compiled kernel and make it downloadable to others I need to make an SD card image of my setup for easy download, so that people can get started easier. Want to Help? # If you want to help me with this, please send me a message on twitter @KenCochrane and also add your name to this Docker issue.\nResources: # LXC: http://raspberrypicloud.wordpress.com/2013/03/12/building-an-lxc-friendly-kernel-for-the-raspberry-pi/ AUFS: http://rpitc.blogspot.sg/p/kernel-rebuild.html Go: http://dave.cheney.net/tag/go-golang-raspberrypi Docker: http://docs.docker.io/en/latest/contributing/devenvironment.html ","date":"May 17, 2013","externalUrl":null,"permalink":"/blog/2013/05/running-docker-on-a-raspberrypi/","section":"Posts","summary":"","title":"Getting Docker up and running on a RaspberryPi","type":"post"},{"content":"","date":"May 17, 2013","externalUrl":null,"permalink":"/tags/go/","section":"Tags","summary":"","title":"Go","type":"tags"},{"content":"","date":"May 17, 2013","externalUrl":null,"permalink":"/tags/linux/","section":"Tags","summary":"","title":"Linux","type":"tags"},{"content":"","date":"May 17, 2013","externalUrl":null,"permalink":"/tags/raspberrypi/","section":"Tags","summary":"","title":"Raspberrypi","type":"tags"},{"content":"","date":"May 17, 2013","externalUrl":null,"permalink":"/tags/raspbian/","section":"Tags","summary":"","title":"Raspbian","type":"tags"},{"content":"","date":"May 12, 2013","externalUrl":null,"permalink":"/tags/rackspace/","section":"Tags","summary":"","title":"Rackspace","type":"tags"},{"content":"Yesterday I wrote a blog post on how to install Docker on to RackSpace Cloud, and one of the steps was to upgrade the kernel to the lastest one so that Docker would be nice and stable. The problem that I found out was that there wasn\u0026rsquo;t much information how to upgrade the kernel on the Rackspace Cloud servers, so I thought I would put the steps here.\nThe goal here is to upgrade Ubuntu 12.04 and 12.10 so that it has the same kernel as 13.04. Here are the steps.\nUbuntu 12.04 # Build an Ubuntu 12.04 server using the \u0026ldquo;Next generation cloud servers\u0026rdquo;, with your desired size. It will give you the password, keep that you will need it later.\nWhen the server is up and running ssh into the server.\n$ ssh root@\u0026lt;server-ip\u0026gt; Once you are logged in you should check what kernel version you are running.\n$ uname -a Linux docker-12-04 3.2.0-38-virtual #61-Ubuntu SMP Tue Feb 19 12:37:47 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux Let\u0026rsquo;s update the server package list\n$ apt-get update Install the 3.8.x kernel using the PPA\n# install the new kernel $ apt-get install linux-generic-lts-raring # update grub so it will use the new kernel after we reboot $ update-grub # update-grub doesn\u0026#39;t always work so lets make sure. ``/boot/grub/menu.lst`` was updated. $ grep 3.8.0- /boot/grub/menu.lst # nope it wasn\u0026#39;t lets manually update ``/boot/grub/menu.lst`` (make sure you are searching for correct kernel version, look at initial uname -a results.) $ sed -i s/3.2.0-38-virtual/3.8.0-19-generic/ /boot/grub/menu.lst # once again lets make sure it worked. $ grep 3.8.0- /boot/grub/menu.lst title Ubuntu 12.04.2 LTS, kernel 3.8.0-19-generic kernel /boot/vmlinuz-3.8.0-19-generic root=/dev/xvda1 ro quiet splash console=hvc0 initrd /boot/initrd.img-3.8.0-19-generic title Ubuntu 12.04.2 LTS, kernel 3.8.0-19-generic (recovery mode) kernel /boot/vmlinuz-3.8.0-19-generic root=/dev/xvda1 ro quiet splash single initrd /boot/initrd.img-3.8.0-19-generic # much better. Reboot server (either via command line or console)\nlogin again and check to make sure the kernel was updated\n$ ssh root@\u0026lt;server_ip\u0026gt; $ uname -a Linux docker-12-04 3.8.0-19-generic #30~precise1-Ubuntu SMP Wed May 1 22:26:36 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux # nice 3.8. Ubuntu 12.10 # Build an Ubuntu 12.10 server using the \u0026ldquo;Next generation cloud servers\u0026rdquo;, with your desired size. It will give you the password, keep that you will need it later.\nWhen the server is up and running ssh into the server.\n$ ssh root@\u0026lt;server-ip\u0026gt; Once you are logged in you should check what kernel version you are running.\n$ uname -a Linux docker-12-10 3.5.0-25-generic #39-Ubuntu SMP Mon Feb 25 18:26:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux Let\u0026rsquo;s update the server package list\n$ apt-get update Install the 3.8.x kernel using the ubuntu-x-swat PPA\n# add the ppa to get the right kernel package $ echo deb http://ppa.launchpad.net/ubuntu-x-swat/q-lts-backport/ubuntu quantal main \u0026gt; /etc/apt/sources.list.d/xswat.list # add the key for the ppa $ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B22AB97AF1CDFA9 # update packages again $ apt-get update # install the new kernel $ apt-get install linux-image-3.8.0-19-generic # make sure grub has been updated. $ grep 3.8.0- /boot/grub/menu.lst title Ubuntu 12.10, kernel 3.8.0-19-generic kernel /boot/vmlinuz-3.8.0-19-generic root=/dev/xvda1 ro quiet splash console=hvc0 initrd /boot/initrd.img-3.8.0-19-generic title Ubuntu 12.10, kernel 3.8.0-19-generic (recovery mode) kernel /boot/vmlinuz-3.8.0-19-generic root=/dev/xvda1 ro quiet splash single initrd /boot/initrd.img-3.8.0-19-generic # looks good. If it doesn\u0026#39;t work for you, look at the notes for 12.04 to fix. Reboot server (either via command line or console)\nlogin again and check to make sure the kernel was updated\n$ ssh root@\u0026lt;server_ip\u0026gt; $ uname -a Linux docker-12-10 3.8.0-19-generic #29~precise2-Ubuntu SMP Fri Apr 19 16:15:35 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux # nice 3.8. ","date":"May 12, 2013","externalUrl":null,"permalink":"/blog/2013/05/upgrading-linux-kernel-ubuntu-rackspace-cloud/","section":"Posts","summary":"","title":"Upgrading the Ubuntu linux kernel on Rackspace cloud","type":"post"},{"content":"I have been playing with Docker a lot lately, and it got me wondering how hard it would be to run Docker on the different Cloud providers. I noticed there were already directions on how to install on Amazon EC2 but nothing for the Rackspace Cloud.\nIf you would like to run Docker on the RackSpace Cloud using Ubuntu you\u0026rsquo;re in luck. I just spent the afternoon figuring out how to get it installed on Ubuntu 12.04, 12.10, and 13.04, and I have included my notes below. 13.04 is the easiest to get up and running since it has the most recent kernel, but the others aren\u0026rsquo;t too bad either, they just need a few more steps, to get them up to par.\nI would love to expand this to other distros on Rackspace, so if you come up with more, send me a note, and so I can link to them.\nUpdate: 05-12-2013 I have updated some information given some feedback by others. Also added a troubleshooting section.\nUbuntu 12.04 # Build an Ubuntu 12.04 server using the \u0026ldquo;Next generation cloud servers\u0026rdquo;, with your desired size. It will give you the password, keep that you will need it later.\nWhen the server is up and running ssh into the server.\n$ ssh root@\u0026lt;server-ip\u0026gt; Once you are logged in you should check what kernel version you are running.\n$ uname -a Linux docker-12-04 3.2.0-38-virtual #61-Ubuntu SMP Tue Feb 19 12:37:47 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux Let\u0026rsquo;s update the server package list\n$ apt-get update Now lets install Docker and it\u0026rsquo;s dependencies. To keep things simple, we will use the Docker install script. It will take a couple of minutes. (see below if you want to install via package)\n$ curl get.docker.io | sudo sh -x Docker runs best with a new kernel, so lets use 3.8.x\n# install the new kernel $ apt-get install linux-generic-lts-raring # update grub so it will use the new kernel after we reboot $ update-grub # update-grub doesn\u0026#39;t always work so lets make sure. ``/boot/grub/menu.lst`` was updated. $ grep 3.8.0- /boot/grub/menu.lst # nope it wasn\u0026#39;t lets manually update ``/boot/grub/menu.lst`` (make sure you are searching for correct kernel version, look at initial uname -a results.) $ sed -i s/3.2.0-38-virtual/3.8.0-19-generic/ /boot/grub/menu.lst # once again lets make sure it worked. $ grep 3.8.0- /boot/grub/menu.lst title Ubuntu 12.04.2 LTS, kernel 3.8.0-19-generic kernel /boot/vmlinuz-3.8.0-19-generic root=/dev/xvda1 ro quiet splash console=hvc0 initrd /boot/initrd.img-3.8.0-19-generic title Ubuntu 12.04.2 LTS, kernel 3.8.0-19-generic (recovery mode) kernel /boot/vmlinuz-3.8.0-19-generic root=/dev/xvda1 ro quiet splash single initrd /boot/initrd.img-3.8.0-19-generic # much better. Reboot server (either via command line or console)\nlogin again and check to make sure the kernel was updated\n$ ssh root@\u0026lt;server_ip\u0026gt; $ uname -a Linux docker-12-04 3.8.0-19-generic #30~precise1-Ubuntu SMP Wed May 1 22:26:36 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux # nice 3.8. Make sure docker is running and test it out.\n$ start dockerd $ docker pull busybox $ docker run busybox /bin/echo hello world hello world Alternate install # If you don\u0026rsquo;t want to run the get.docker.io script and want to use packages instead, you can use the docker PPA. Here is how you use it. Replace step 5 with the following 3 steps.\nAdd the custom package sources to your apt sources list. Copy and paste the following lines at once. $ sudo sh -c \u0026#34;echo \u0026#39;deb http://ppa.launchpad.net/dotcloud/lxc-docker/ubuntu precise main\u0026#39; \u0026gt;\u0026gt; /etc/apt/sources.list\u0026#34; Update your sources. You will see a warning that GPG signatures cannot be verified. $ sudo apt-get update Now install it, you will see another warning that the package cannot be authenticated. Confirm install. $ apt-get install lxc-docker Ubuntu 12.10 # Build an Ubuntu 12.10 server using the \u0026ldquo;Next generation cloud servers\u0026rdquo;, with your desired size. It will give you the password, keep that you will need it later.\nWhen the server is up and running ssh into the server.\n$ ssh root@\u0026lt;server-ip\u0026gt; Once you are logged in you should check what kernel version you are running.\n$ uname -a Linux docker-12-10 3.5.0-25-generic #39-Ubuntu SMP Mon Feb 25 18:26:58 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux Let\u0026rsquo;s update the server package list\n$ apt-get update Now lets install Docker and it\u0026rsquo;s dependencies. To keep things simple, we will use the Docker install script. It will take a couple of minutes.\n$ curl get.docker.io | sudo sh -x Docker runs best with a new kernel, so lets use 3.8.x\n# add the ppa to get the right kernel package $ echo deb http://ppa.launchpad.net/ubuntu-x-swat/q-lts-backport/ubuntu quantal main \u0026gt; /etc/apt/sources.list.d/xswat.list # add the key for the ppa $ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 3B22AB97AF1CDFA9 # update packages again $ apt-get update # install the new kernel $ apt-get install linux-image-3.8.0-19-generic # make sure grub has been updated. $ grep 3.8.0- /boot/grub/menu.lst title Ubuntu 12.10, kernel 3.8.0-19-generic kernel /boot/vmlinuz-3.8.0-19-generic root=/dev/xvda1 ro quiet splash console=hvc0 initrd /boot/initrd.img-3.8.0-19-generic title Ubuntu 12.10, kernel 3.8.0-19-generic (recovery mode) kernel /boot/vmlinuz-3.8.0-19-generic root=/dev/xvda1 ro quiet splash single initrd /boot/initrd.img-3.8.0-19-generic # looks good. If it doesn\u0026#39;t work for you, look at the notes for 12.04 to fix. Reboot server (either via command line or console)\nlogin again and check to make sure the kernel was updated\n$ ssh root@\u0026lt;server_ip\u0026gt; $ uname -a Linux docker-12-10 3.8.0-19-generic #29~precise2-Ubuntu SMP Fri Apr 19 16:15:35 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux # nice 3.8. Make sure docker is running and test it out.\n$ start dockerd $ docker pull busybox $ docker run busybox /bin/echo hello world hello world Ubuntu 13.04 # Build an Ubuntu 13.04 server using the \u0026ldquo;Next generation cloud servers\u0026rdquo;, with your desired size. It will give you the password, keep that you will need it later.\nWhen the server is up and running ssh into the server.\n$ ssh root@\u0026lt;server-ip\u0026gt; Once you are logged in you should check what kernel version you are running.\n$ uname -a Linux docker-1304 3.8.0-19-generic #29-Ubuntu SMP Wed Apr 17 18:16:28 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux Let\u0026rsquo;s update the server package list\n$ apt-get update Now lets install Docker and it\u0026rsquo;s dependencies. To keep things simple, we will use the Docker install script. It will take a couple of minutes.\n$ curl get.docker.io | sudo sh -x Make sure docker is running and test it out.\n$ start dockerd $ docker pull busybox $ docker run busybox /bin/echo hello world hello world What\u0026rsquo;s Next # Now that you have Docker running on a server, you can look at the different Docker examples in the documentation to see how it works, and then build something, and let everyone know what you have built. If you have any issues or suggestions, open a github issue and let everyone know. Docker is a new project, and it is moving quick, so any suggestions that you have might help shape the future of the project.\nTrouble shooting # If you are pulling a repo and you get an error like this.\nError: exit status 1: bsdtar: Linkname can\u0026#39;t be converted from UTF-8 to current locale. bsdtar: Linkname can\u0026#39;t be converted from UTF-8 to current locale. It means the the docker daemon doesn\u0026rsquo;t have the correct locales loaded on startup. To fix it make sure your init script looks something like this.\nMake sure the path to the docker binary is correct because in some installs it might be /usr/local/bin and others /usr/local/\ndescription \u0026#34;Run docker\u0026#34; start on runlevel [2345] stop on starting rc RUNLEVEL=[016] respawn script test -f /etc/default/locale \u0026amp;\u0026amp; . /etc/default/locale || true LANG=$LANG LC_ALL=$LANG /usr/local/bin/docker -d end script ","date":"May 11, 2013","externalUrl":null,"permalink":"/blog/2013/05/running-docker-on-rackspace-cloud/","section":"Posts","summary":"","title":"Running Docker on Rackspace with Ubuntu","type":"post"},{"content":"","date":"February 19, 2013","externalUrl":null,"permalink":"/tags/djangomaine/","section":"Tags","summary":"","title":"Djangomaine","type":"tags"},{"content":"I recently gave a talk at a DjangoMaine meetup introducing folks to Python and Django. There wasn\u0026rsquo;t enough time to go over everything, so I tried to pick and choose the parts that I felt where important to go over, in order to have a good overview of what Python and Django is all about.\nI hope this was useful to the folks that attended. If so, it would be really helpful to know what parts you would like to know more about. Then we can have some follow up talks doing a deeper dive into those select topics.\nFuture Tutorials # Also, if there is a demand, I\u0026rsquo;m willing to do a half or whole day Python and Django tutorial, where we would be able to take the time needed to go into all the different topics, so that you would have a good understanding of each one, when you left. Ideally I would probably break up the sessions into beginner, intermediate, and advanced so that people can move to the different levels as they get more familiar and comfortable with Django.\nI would also like to offer a talk to local high schools. If we have enough kids that are interested, I\u0026rsquo;d be willing to go to your high school and give a talk and introduce these topics to the kids.\nIf you are interested in learning more about Django, please let me know. I don\u0026rsquo;t want to waste my time doing all the work preparing for a tutorial, if there isn\u0026rsquo;t enough demand.\nSlides # We didn\u0026rsquo;t record the talk, but you can find my slides below.\nDownload # The slides are also available to download in pdf form: Intro to Python and Django Presentation slides\nIf you have any questions, or comments, feel free to contact me via the comments below, or on Twitter (@KenCochrane)\n","date":"February 19, 2013","externalUrl":null,"permalink":"/blog/2013/02/intro-to-python-and-django-presentation/","section":"Posts","summary":"","title":"Intro to Python and Django Presentation","type":"post"},{"content":"I have recently changed my Django based blog to a statically generated blog based on Pelican, and hosted by GitHub Pages. This is for a couple of reasons.\nThe first reason was because my blog is really simple, it was just a bunch of reStructuredText documents that were converted to HTML. I didn\u0026rsquo;t need the Django-admin features and I always felt it was a little overkill for my blog.\nThe second reason was because I wanted to host my blog contents on GitHub so that others could change/update/edit my blog posts and send me pull requests if they have something good to add. Doing this with my old blog would have been hard, but it will be really easy with this current setup.\nI was looking through all of the different static generators, and boy there are a lot of them. I decided to pick pelican because it did what I needed, and was real easy to setup and use.\nMy requirements for a site generator:\nPython: I wanted it python based, in case it needed any changes, I wouldn\u0026rsquo;t need much of a learning curve learning a new language. Open Source: I wanted to have the ability to change it to add my own features in case it doesn\u0026rsquo;t have them to start. Plus Open Source code rules. reStructureText: I wanted to write my blog posts in ReST. Themes: I wanted some out of the box themes that I could pick from and customize Easy: I wanted something with minimum setup and overhead to get started. Maintain Links: I wanted to keep the same URL structure from my old blog so I wouldn\u0026rsquo;t get lots of 404 errors once I converted. Active development: I wanted a project that is currently under active development. After I did some quick searches, I found Pelican, and I liked what I saw. I wrote a simple python script that pull out my blog posts from my old blog and generated the initial ReST docs, which got me most of the way there.\nOnce I got the content pulled out, I just needed to pick a theme, and configure the settings. Then setup my github pages site, and push all the code. The last change was to change my DNS records, and then I was done.\nNow anytime I want to write a blog post, all I have to do is add a new ReST doc and rerun the build command, and commit my changes, and push my repo. Then github automatically updates my site. Simple as that.\nI\u0026rsquo;ll run it from github for a little while and see how I like it. I might end up changing hosting, since github has very limited features, and doesn\u0026rsquo;t allow you to add any rewrite rules or anything custom. I could have just as easily hosted on AWS s3, or dotCloud, but I wanted to try this out first since it was pretty easy to setup.\nIf you switched to a static site generator, or are thinking about doing it, let me know what you think and what tools you are using.\n","date":"December 9, 2012","externalUrl":null,"permalink":"/blog/2012/12/new-pelican-based-blog/","section":"Posts","summary":"","title":"New Pelican Based Blog","type":"post"},{"content":"","date":"December 9, 2012","externalUrl":null,"permalink":"/tags/pelican/","section":"Tags","summary":"","title":"Pelican","type":"tags"},{"content":"As I mentioned in a recent blog post, in the 11 months since I wrote my first post on dotCloud, I now work there. Besides me working there, there has been a lot of other changes at dotCloud, and I wanted to take the time to update my original post so that it was up to date and had all the recent information. I\u0026rsquo;m going to completely rewrite the old blog post here, with updated information, and leave the old one around for posterity.\ndotCloud\u0026rsquo;s goal is to provide a bunch of different independent services that you can use as building blocks to build your application. If you need a database, take your pick from one of the many they support. Need an application that uses Django or Rails on the front end, and Java in the backend, that is fine, you can do that too. They realize that most developers don\u0026rsquo;t stick to one standard technology stack for all of their applications, so this allows you the flexibility to use the best tool for the job. It also gives you a nice playground to try out new services and see how they run without having to install, configure and maintain the service just for testing.\nI\u0026rsquo;m going to go over the steps that it took to install my blog onto dotCloud, and hopefully answer some common questions along the way.\nDocumentation # Before I get started with any new service the first thing I usually do is look at the documentation. DotCloud has a nice list of documents along with some tutorials on how to get started. These 4 documents were the ones that I used the most.\nhttp://docs.dotcloud.com/firststeps/platform-overview/\nhttp://docs.dotcloud.com/tutorials/python/django/\nhttp://docs.dotcloud.com/services/mysql/\nhttp://docs.dotcloud.com/services/mysql-masterslave/\nFirst Steps: # Like all cool services these days, dotCloud uses a python based CLI, so before we can get started we need to install the dotCloud client and configure it so that we can start using it.\n# create my dotcloud virtual environment. $ mkvirtualenv dotcloud # install dotcloud client using pip $ pip install dotcloud # create our application called blog $ dotcloud create blog #enter api key that we got from: http://www.dotcloud.com/account/settings when prompted #\u0026lt;key goes here\u0026gt; # if you were not prompted to enter your key you can run this command, and it will let you enter your API key again. $ dotcloud register Now that we have the client all setup, and an application created, now we can start building our service. I have forked my blog repository on github so that I could make dotCloud specific changes to it and not effect my original repo.\n# go into projects directory cd ~/projects # forked kencochranenet to kencochranenet_dotcloud, now clone that. locally git clone git://github.com/kencochrane/kencochranenet_dotcloud.git kencochranenet_dotcloud # go into the new directory. cd kencochrane_dotcloud Reading through the documentation tells me that I need to create a wsgi.py file and put in the root of my project. Using http://docs.dotcloud.com/tutorials/python/django/#wsgi-py as a template, I created my wsgi.py file below. I had issues with the default template and I needed to add a directory to the sys.path so that wsgi could find my django apps correctly. Here is my finished file.\nimport os import sys # Ken added this, only thing that is different from the example template (not counting settings file name) sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__),\u0026#39;mysite\u0026#39;))) os.environ[\u0026#39;DJANGO_SETTINGS_MODULE\u0026#39;] = \u0026#39;mysite.settings\u0026#39; import django.core.handlers.wsgi djangoapplication = django.core.handlers.wsgi.WSGIHandler() def application(environ, start_response): if \u0026#39;SCRIPT_NAME\u0026#39; in environ: del environ[\u0026#39;SCRIPT_NAME\u0026#39;] return djangoapplication(environ, start_response) DotCloud uses PIP requirements files to manage your project dependencies. We already have our pip requirements file where it needs to be and named correctly so we don\u0026rsquo;t need to do anything, but if we didn\u0026rsquo;t have one, we would need to create one and put it in the root, and call it requirements.txt\nServices # When we add a service to our deployment stack, dotCloud gives us the appropriate connection information in a file called \u0026lsquo;/home/dotcloud/environment.json\u0026rsquo; that is available to us on our deployment container. This allows us to not have to hard code username/password and server urls in our settings.py file, and it also makes it a little more secure for us since we don\u0026rsquo;t have to have that info in our source repository.\nThis is how we use it. At the top of your settings.py file you will need to add the following. snippet.\nimport json with open(\u0026#39;/home/dotcloud/environment.json\u0026#39;) as f: env = json.load(f) Once we have that added to the settings.py file, we now have a variable env that has all of the env settings we need.\nYou could go a little further add some custom code to check if the environment.json file exists, and if it does, you know you are in production, so use that setup, or if not, then you must be in local mode, so use your local settings. If you want to get really cool, you can have your own json file that has a similar setup for local development, and if it doesn\u0026rsquo;t find the dotcloud one, it could look for your own, and load your settings from that. This will allow you to use the same settings file for both production and dev, with only a little bit of code at the top to load the correct env file.\nDatabase # Most applications need a database, and this blog is no different. This is how we setup our database to work with our blog on dotcloud. We are going to be using mysql for our database. With Django you need to set your database settings in your settings.py. This is how we setup a mysql database connection inside of our settings.py. Notice that the name of the database doesn\u0026rsquo;t come from the env, you pick that yourself.\nDATABASES = { \u0026#39;default\u0026#39;: { \u0026#39;ENGINE\u0026#39;: \u0026#39;django.db.backends.mysql\u0026#39;, \u0026#39;NAME\u0026#39;: \u0026#39;blogdb\u0026#39;, \u0026#39;USER\u0026#39;: env[\u0026#39;DOTCLOUD_DB_MYSQL_LOGIN\u0026#39;], \u0026#39;PASSWORD\u0026#39;: env[\u0026#39;DOTCLOUD_DB_MYSQL_PASSWORD\u0026#39;], \u0026#39;HOST\u0026#39;: env[\u0026#39;DOTCLOUD_DB_MYSQL_HOST\u0026#39;], \u0026#39;PORT\u0026#39;: int(env[\u0026#39;DOTCLOUD_DB_MYSQL_PORT\u0026#39;]), } } Create the Database # dotCloud gives you your own dedicated database, with full root access. With great power comes great responsibilities. One of those responsibilities is that you need to create your own database schemas, and users yourself. Which means you normally need to do something like this.\n# connect to dotcloud mysql server instance $ dotcloud run blog.db -- mysql -u root -p # mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \\g. Your MySQL connection id is 34 Server version: 5.1.41-3ubuntu12.10 (Ubuntu) # create the user and database and give user permissions to database. Type \u0026#39;help;\u0026#39; or \u0026#39;\\h\u0026#39; for help. Type \u0026#39;\\c\u0026#39; to clear the current input statement. mysql\u0026gt; create database blogdb; Query OK, 1 row affected (0.00 sec) mysql\u0026gt; create user \u0026#39;blog_username\u0026#39; identified by \u0026#39;strong_password\u0026#39;; Query OK, 0 rows affected (0.05 sec) mysql\u0026gt; grant all on blogdb.* to \u0026#39;blog_user\u0026#39;@\u0026#39;%\u0026#39;; Query OK, 0 rows affected (0.04 sec) mysql\u0026gt; flush privileges; Query OK, 0 rows affected (0.00 sec) mysql\u0026gt; exit;Bye Shared connection to database closed. Does that look familiar? I have it here in case you want to do it the long way.\nTo make things easier, we are going to create a small python script that will check to see if we have our database created, and if not, it will create it for us. This will make it so that we don\u0026rsquo;t have to login into our database and do it by hand before we deploy. The file is called createdb.py and this is what it looks like. This script is for mysql. If you want a postgreSQL database, you can use this as a template and change it so that it will work with postgreSQL.\nimport MySQLdb import os from wsgi import * def create_dbs(names): print(\u0026#34;create_dbs: let\u0026#39;s go.\u0026#34;) django_settings = __import__(os.environ[\u0026#39;DJANGO_SETTINGS_MODULE\u0026#39;], fromlist=\u0026#39;DATABASES\u0026#39;) print(\u0026#34;create_dbs: got settings.\u0026#34;) databases = django_settings.DATABASES for name, db in databases.iteritems(): if name in names and db[\u0026#39;ENGINE\u0026#39;].endswith(\u0026#39;mysql\u0026#39;): host = db[\u0026#39;HOST\u0026#39;] user = db[\u0026#39;USER\u0026#39;] password = db[\u0026#39;PASSWORD\u0026#39;] port = db[\u0026#39;PORT\u0026#39;] db_name = db[\u0026#39;NAME\u0026#39;] print \u0026#39;creating database %s on %s\u0026#39; % (db_name, host) db = MySQLdb.connect(user=user, passwd=password, host=host, port=port) cur = db.cursor() print(\u0026#34;Check if database is already there.\u0026#34;) cur.execute(\u0026#34;\u0026#34;\u0026#34;SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = %s\u0026#34;\u0026#34;\u0026#34;, (db_name,)) results = cur.fetchone() if not results: print(\u0026#34;Database %s doesn\u0026#39;t exist, lets create it.\u0026#34; % db_name) sql = \u0026#34;\u0026#34;\u0026#34;CREATE DATABASE IF NOT EXISTS %s \u0026#34;\u0026#34;\u0026#34; % (db_name,) print(\u0026#34;\u0026gt; %s\u0026#34; % sql) cur.execute(sql) print(\u0026#34;.....\u0026#34;) else: print(\u0026#34;database already exists, moving on to next step.\u0026#34;) if __name__ == \u0026#39;__main__\u0026#39;: import sys print(\u0026#34;create_dbs start\u0026#34;) create_dbs(sys.argv[1:]) print(\u0026#34;create_dbs all done\u0026#34;) Adding a cache # Since most of the blog content doesn\u0026rsquo;t change much, it is a great candidate for caching. So we are going to take advantage of Django\u0026rsquo;s built in caching abilities and add some caching to our blog. Normally I use memcached for my caching, but dotCloud\u0026rsquo;s memcached support is limited right now. The reason why it is limited is because memcached doesn\u0026rsquo;t have any built in authentication mechanism, and in order to make sure it is secure, you need to run a special version of memcached that supports SASL, and most of the memcached clients don\u0026rsquo;t support this. So instead of deploying an insecure service, they decided to not support it fully. There are ways to use it, but it involves all kind of complicated firewall rules and running something like stunnel. So it is possible, but it isn\u0026rsquo;t very clean.\nInstead they recommend that you use redis instead, redis has the same caching abilities that memcached has, plus a lot more, including authentication. So we are going to use redis for our cache. In order to use redis, we will need to add the redis library because redis caching support isn\u0026rsquo;t built into Django. In your requirements.txt file you will need to add django-redis==1.4.5 so that the libraries will be available for Django to use.\nOnce you have the library installed, you will need to add these settings to your settings.py file so that django knows which redis server and password to use.\nCACHES = { \u0026#39;default\u0026#39;: { \u0026#39;BACKEND\u0026#39;: \u0026#39;redis_cache.cache.RedisCache\u0026#39;, \u0026#39;LOCATION\u0026#39;: env[\u0026#39;DOTCLOUD_CACHE_REDIS_HOST\u0026#39;]+\u0026#39;:\u0026#39;+env[\u0026#39;DOTCLOUD_CACHE_REDIS_PORT\u0026#39;], \u0026#39;OPTIONS\u0026#39;: { \u0026#39;DB\u0026#39;: 1, \u0026#39;PASSWORD\u0026#39;: env[\u0026#39;DOTCLOUD_CACHE_REDIS_PASSWORD\u0026#39;], \u0026#39;PARSER_CLASS\u0026#39;: \u0026#39;redis.connection.HiredisParser\u0026#39; }, }, } # we also are going to use redis for our session cache as well. SESSION_ENGINE = \u0026#39;django.contrib.sessions.backends.cached_db\u0026#39; For more information about using redis as your cache for Django, check out these links.\nhttps://github.com/niwibe/django-redis http://pypi.python.org/pypi/django-redis/1.4.5 https://docs.djangoproject.com/en/1.3/topics/cache/ http://answers.dotcloud.com/question/213/redis-cache-settings-for-django Django Admin # We also need an easy way to create our django admin account for us. In order to do that I have this mkadmin.py script. This will default the password to \u0026lt;\u0026lsquo;P@s$w0rd1\u0026rsquo;\u0026gt;, once you have your code installed, you will need to login to the admin account and change your password to something more secure.\n#!/usr/bin/env python from wsgi import * from django.contrib.auth.models import User u, created = User.objects.get_or_create(username=\u0026#39;admin\u0026#39;) if created: u.set_password(\u0026#39;P@s$w0rd1\u0026#39;) u.is_superuser = True u.is_staff = True u.save() Media # We need to put our static and media files in the following locations: static=/home/dotcloud/data/static/ and media=/home/dotcloud/data/media/. Because of this we need to make sure we change our settings.py file, and setup an nginx.conf file to map to the correct locations. Here are the settings.py file changes.\n# media settings MEDIA_ROOT = \u0026#39;/home/dotcloud/data/media/\u0026#39; MEDIA_URL = \u0026#39;/media/\u0026#39; # static settings STATIC_ROOT = \u0026#39;/home/dotcloud/data/static/\u0026#39; STATIC_URL = \u0026#39;/static/\u0026#39; # admin prefix ADMIN_MEDIA_PREFIX = \u0026#39;/static/admin/\u0026#39; Here is the nginx.conf\nlocation /media/ { root /home/dotcloud/data ; } location /static/ { root /home/dotcloud/data ; } Post Install # We are going to create a postinstall script to handle all of the tasks we need to do after we install our code on the server. This is what will call our createdb.py, and mkadmin.py files from above, as well as syncing our database, running migrations and running collectstatic to move all static files into the right locations.\n#!/bin/sh python createdb.py default python mysite/manage.py syncdb --noinput python mysite/manage.py migrate python mkadmin.py mkdir -p /home/dotcloud/data/media /home/dotcloud/data/static python mysite/manage.py collectstatic --noinput Don\u0026rsquo;t forget to make sure your postinstall, createdb.py and mkadmin.py scripts are executable.\n# make the script executable. $ chmod +x postinstall createdb.py mkadmin.py dotcloud.yml # Now that we have our application\u0026rsquo;s project structure all setup and configured the way dotCloud wants it, we can configure our deployment stack. This is done with a file called dotcloud.yml. For more information about the dotcloud.yml file check out this link: http://docs.dotcloud.com/guides/build-file/\nwww: type: python db: type: mysql cache: type: redis This is telling us that we want 3 services, a python www service, a mysql db service, and a redis cache service. This is a very basic setup, and you can get a lot more complicated depending on what you want to achieve. Notice that this isn\u0026rsquo;t setup for high availability because none of the instances are scaled. See the section about scaling below for more information. If you are running in a production app on dotCloud it is recommended that you scale all of your services so that they can withstand EC2 server crashes, and other unforeseen issues.\nDeployment # Now we are ready to deploy our Django app, but before I go any further it is important to know the following. Dotcloud will pay attention to your .gitignore files. If you have a settings file in your .gitignore file so that it doesn\u0026rsquo;t get saved in the repo, it will not push those changes up to the cloud. You will need to remove it from the .gitignore in order to get those files out there. It is also import to remember that only changes that are committed are pushed, so don\u0026rsquo;t forget to commit your changes. If you wanted to be tricky you could use a post install script to pull down the file from a secure location and install it that way, if you want to make things super secure.\nEverything is all setup, so all we have to do is push our application to dotCloud.\n# push out your changes to the server $ dotcloud push blog . Service info # Once you push your code to dotCloud you can see what it looks like by running the info command.\n# get the information about our new services $ dotcloud info blog cache: config: redis_password: \u0026lt;password\u0026gt; redis_replication: true instances: 1 type: redis db: config: mysql_masterslave: true mysql_password: \u0026lt;password\u0026gt; instances: 1 type: mysql www: config: static: static uwsgi_processes: 4 instances: 1 type: python url: \u0026lt;url was here\u0026gt; Scaling # Scaling is the ability to grow your application so that it can handle more traffic, or possible failures that might occur. With a normal non PaaS setup, scaling an application can be quite painful and time consuming, but with a PaaS it can be as easy as running a few commands. There are three types of scaling, Vertical, Horizontal, High Availability.\nVertical scaling, means growing the service you have now so that it can get bigger. This is popular with databases, the bigger a database gets the more space and memory it needs.\nHorizontal scaling means creating more then one instance of a service so it spread the work between the different services, giving you greater capacity.\nHigh Availability means that you have more then one service running at a time, so that if one of the services has an issue, the other one will pick up the slack. This will help avoid downtime, when failures occur (EC2 instance crashes). Ideally when running in production, all of your services should be scaled for High Availability.\nThere are two kinds of services, stateful, and stateless. Stateful services are services that holds persistent data. Examples of stateful services are mysql, redis, postgresql, solr, MongoDB and RabbitMQ.\nHorizontally High availability scaling a stateful service on dotCloud means creating a master/slave setup, which can switch the slave with the master automatically if the master has any issues. dotCloud supports HA scaling on MySQL, redis, and MongoDB.\nStateful services scale like this\nmysql : 2 (master/slave) redis : 2 (master/slave) mongodb : 3 or 5 (using replica sets) Scaling a stateless and one of the supported stateful services is the same. You would just run the scale command line command.\n$ dotcloud scale app db=2 For stateless applications, you are limited to a set number of scaling units, unless you are on the enterprise plan. If you need to have an application with lots of scale units, you should contact dotCloud, and let them know what you are planning to do, and they will advise you on how best to accomplish your goals.\nLink: http://docs.dotcloud.com/guides/scaling/\nDatabase Backups # Just because you are hosting your application on dotCloud doesn\u0026rsquo;t mean you shouldn\u0026rsquo;t backup your data. The most important data to backup is your database. Luckily dotCloud makes it easy to back up your database. There is a very helpful guide on how to setup your database backups here: http://docs.dotcloud.com/guides/backups/\nEmail # If you need to send or receive email from your application, you can do that to. Because dotCloud runs on EC2, and EC2 is a popular place where SPAMMERS send SPAM from, it is best to use a 3rd party email provider to send your emails for you. Popular ones are MailGun, SendGrid, CritSend, and Amazon SES.\nYou can set this up a couple of different ways. The first way is the easiest way possible, it allows you to configure the SMTP settings for each service. You would do it like this (see below). You can manually set the smtp settings for that service, and when your application needs to send an email it will use those settings. This is the most simple setup, but there are downsides to this approach. You would need to set this for each service, if you have more then one that will be duplicated everywhere. Also if you want to change your settings, you will need to destroy your service and recreate it, since those configs can only be set once when the service is created.\nwww: type: python config: smtp_server: smtp.mailgun.org smtp_port: 25 smtp_username: postmaster@company.com smtp_password: YourMailGunPassword A better approach would be to use dotCloud\u0026rsquo;s SMTP service. The SMTP service is built to receive emails from your services and forward them to the appropriate location. It is best to use a 3rd party email provider, but you can also use the typical poor mans solution, where you use gmail to send your emails. Be careful when using gmail, because you aren\u0026rsquo;t aloud to send a lot of emails via gmail, once you hit your daily limit you will be blocked, so this is fine for a few emails a day, don\u0026rsquo;t trust it for everyday stuff. Also, the emails will always be coming from your gmail address, fine for system emails, but not if you are trying to run a legit business.\nHere is an example using mailgun.\nmailer: type: smtp config: smtp_relay_server: smtp.mailgun.org smtp_relay_port: 587 smtp_relay_username: postmaster@yourmailgundomain.com smtp_relay_password: YourMailgunPassword Here is an example using gmail.\nmailer: type: smtp config: smtp_relay_server: smtp.gmail.com smtp_relay_port: 587 smtp_relay_username: your_gmail_username@gmail.com smtp_relay_password: Your_Gmail_Password Once you have these all setup, they will be available in your environment.json file.\nIf you want to receive email, it is best to use a service like MailGun .\nLinks:\nhttp://docs.dotcloud.com/guides/emails/ http://docs.dotcloud.com/services/smtp/ Cron jobs # If your app needs to run cron jobs, follow the steps in this guide: http://docs.dotcloud.com/guides/periodic-tasks/\nCelery # This blog doesn\u0026rsquo;t really have a need for celery, but dotCloud does support it. For more information follow this link: http://docs.dotcloud.com/tutorials/python/django-celery/\nS3FS # If you store data on s3 you can mount your s3 bucket so that you can have access to s3 from your application, just like it was a local directory on your container. This is helpful for storing files that are uploaded by your visitors, or to share files between different web processes. Follow these instructions to set it up: http://docs.dotcloud.com/guides/s3fs/\nLogs # If you need to look at the logs to see how it is going you can do it two ways. The first way will tail your logs for you to your console.\n# look at logs of your service, it will tail them to your console. ctrl-c to stop. $ dotcloud logs blog.www Or login via ssh and look at your logs.\n# Open up a shell $ dotcloud ssh blog.www Here are the ones you most likely care about.\n# nginx access and error logs. /var/log/nginx/\u0026lt;app_name\u0026gt;.{access,error}.log # wsgi error logs /var/log/supervisor/uswgi.log Restart Service # If you need to restart your service just issue this command.\n# restart the service dotcloud restart blog.www Links # Read how this service stacks up against other services like it in my Django hosting roundup http://www.dotCloud.com ","date":"March 23, 2012","externalUrl":null,"permalink":"/blog/2012/03/deploying-a-django-application-on-dotcloud/","section":"Posts","summary":"","title":"Deploying my Django application to dotCloud: Part 2","type":"post"},{"content":"","date":"March 23, 2012","externalUrl":null,"permalink":"/tags/deployment/","section":"Tags","summary":"","title":"Deployment","type":"tags"},{"content":"A lot has happened in the 11 months since I wrote my first blog article on dotCloud. There was an Egyptian revolution, a royal wedding, a new iPhone and iPad, Osama bin Laden was killed, and on a more personal note, I left my job at CashStar and I now work for dotCloud. In the grand scheme of things, my job change was nothing compared to those big events of the past 11 months, but it was pretty big for me.\nI had worked for CashStar.com for over 3.5 years, and I was there from pretty much the beginning, so it was hard for me to walk away. When I started doing research on PaaS\u0026rsquo;s over a year ago, I did it so that I could learn more about how these systems were built, so that I could take that knowledge and bring it to CashStar to improve our infrastructure there.\nA funny thing happened as I was doing my research, I started to really love the technology that went into making a PaaS, and over time, I thought that it was such a cool concept, that I wanted to know everything there was to know about them, so that I could build my own. After trying out all of the PaaS\u0026rsquo;s that I could find, and learning and blogging about them. I decided I would see what it would take to build my own PaaS. The first thing I realized is that there is a lot more going on under the covers then one might imagine, and this wasn\u0026rsquo;t going to be an easy problem to solve. If it was easy, then everyone and their brother would have one, but they don\u0026rsquo;t. There are a lot of PaaS\u0026rsquo;s out there now, but if you think about it, that is nothing compared to the number of web hosting companies.\nAfter realizing that this isn\u0026rsquo;t something I was going to be able to build in a weekend, I kind of lost motivation. I have 2 kids and a job that was making me work a ton of hours, so the few extra hours I had in a day where spent hanging out with my family, and what ever was left over was spent playing one of my many sports, or getting some much needed sleep. So, for a while my dream of making my own PAAS faded for a little while, but it never disappeared.\nAs fate would have it, a few months later I ended up talking with the dotCloud guys, and next thing you know they made me an offer to join their company. After talking it over with my wife, I decided to go for it. Since dotCloud is based in San Francisco, I\u0026rsquo;ll be working from home in Maine, and telecommuting to work each day from my home office. I have been working for dotCloud for about 5 weeks so far, and it has been awesome. I have learned so many new and cool things in the past month, it has been great. The dotCloud team is top notch and I can\u0026rsquo;t wait to learn even more from them.\nMy title at dotCloud is Site Reliability Engineer. You might have never heard about that title before. To be honest I had never heard of it before myself. My primary goal is to make sure that the dotCloud platform is as stable and reliable as it could be. One of our goals is to automate everything, so that if something fails, we have a process that will notice this, and auto correct it for us. Since dotCloud is build on top of Amazon EC2, things can happen at anytime, and they do. So when they happen, we need a self healing platform that will fix itself when things break in the middle of the night. If it doesn\u0026rsquo;t fix it self, someone will need to be woken up to fix it, and no one wants that.\nAnother one of my goals is to make sure that dotCloud is the best developers platform available. I want to make it so that developers can do what ever they want to do, and if they have any issues, give them the tools they need to solve their own problems. If they get stuck, we will be there to help them through their problems, via IRC, or email support.\nEvery developer writes code with bugs, and if you find one that tells you otherwise, they are lying. dotCloud allows developers to see all of their application and system logs easily, and also gives them direct SSH access to the containers where their code is running so that they have the ability to see why something isn\u0026rsquo;t working. Most PaaS\u0026rsquo;s don\u0026rsquo;t provide this, which makes it harder to figure out why the code that you wrote isn\u0026rsquo;t working the way that you had planned.\nOne of the cool things that dotCloud does is that they are heavy believers in dog fooding. They encourage employees to use the platform as much as possible, and it is amazing to know that a lot of dotCloud runs on dotCloud. This helps smooth out all of the different developer pain points, and makes for a better platform overall.\nThey also listen to what their users have to say, and take it to heart. If you have used dotCloud in the past, and were turned off for some reason, you should come back and try it again, it has changed a ton, and the issue that you might have had a while ago, might no longer be there. If you are still having issues, please tell us, because if we don\u0026rsquo;t know that it is an issue for you, there is no way we can fix it.\nOne of the first things that I noticed when I joined was the support that they offer, it is unbelievable. I get an email for every single support ticket that is created, and the responses. I have learned a lot just by reading those emails. We get tickets for everything from sticker requests all the way to \u0026ldquo;my website is having issues, can you help\u0026rdquo;. Each day there is at least one engineer dedicated to answering support tickets, and that is all they do for the day, everything else takes a back seat. The same people who are building dotCloud are the ones answering your questions, so when you get an answer you know it is coming directly from the source. Most tickets are responded to and closed really quickly. It wasn\u0026rsquo;t always this good, but it is something they work on improving all of the time, because they realize, that it is important to have a quick and helpful support staff around to solve your problems when you have them.\nNow that I\u0026rsquo;m on the inside, I can see and hear about all of the cool new things they are working on for the future, and I\u0026rsquo;m really excited. If you are a developer and you haven\u0026rsquo;t tried dotCloud yet, or if you have tried them a while ago and you haven\u0026rsquo;t tried them recently you should check them out. What are you waiting for it, is FREE, it won\u0026rsquo;t cost you anything to try it out, and you never know it might change your life.\nOne of the sad things about working for dotCloud will mean that I won\u0026rsquo;t be able to blog about the PaaS industry that much anymore. I\u0026rsquo;m going to try and avoid doing it, due to the conflict of interest. I will keep around my old blog post so that everyone can benefit, and I\u0026rsquo;ll try to keep them up to date, but besides that I won\u0026rsquo;t be writing about anything unless it is really cool.\nIf you have tried out dotCloud, let me know what you think, and let me know if there is anything we can do to make it better for you.\n","date":"March 23, 2012","externalUrl":null,"permalink":"/blog/2012/03/im-now-working-for-dotcloud/","section":"Posts","summary":"","title":"I'm now working for dotCloud","type":"post"},{"content":"","date":"March 23, 2012","externalUrl":null,"permalink":"/tags/job/","section":"Tags","summary":"","title":"Job","type":"tags"},{"content":"","date":"March 23, 2012","externalUrl":null,"permalink":"/tags/paas/","section":"Tags","summary":"","title":"Paas","type":"tags"},{"content":"Sentry is a realtime event logging and aggregation platform. At it’s core it specializes in monitoring errors and extracting all the information needed to do a proper post-mortum without any of the hassle of the standard user feedback loop.\nThe main feature of sentry and the ability to send all of your application logs to one place, and then aggregate them, so that you only get one error email for the same error. This will keep your mailbox from flooding, when something goes wrong.\nPutting your logging server on a different server or network then your production servers is a good idea. If something goes wrong, and you can\u0026rsquo;t access your servers, you can still see what errors were getting thrown before the servers started having problems.\nFollow these easy steps to get sentry up and running on DotCloud.\nCreate a place to store your project $ mkdir -p ~/projects Go into the projects directory $ cd ~/projects Clone git repo from github, requires git client $ git clone git://github.com/kencochrane/sentry-on-dotcloud.git Go into the new project directory $ cd sentry-on-dotcloud Creating the virtualenv (using virtualenvwrapper, virtualenv, and pip) $ mkvirtualenv --no-site-packages --distribute sentry-on-dotcloud Install all of the Sentry requirements via pip and the requirements.txt file. $ pip install -r requirements.txt Installing the dotCloud client http://docs.dotcloud.com/firststeps/install/ (here are the steps for Linux and Mac OSX) $ sudo pip install -U dotcloud Sign up for a dotcloud account https://www.dotcloud.com/accounts/register/ if you haven\u0026rsquo;t already. The first time you use the dotCloud account you will need to add your api key. So type dotcloud and follow the steps. You can find your API key at http://www.dotcloud.com/account/settings $ dotcloud Create your dotcloud application $ dotcloud create sentry Change the SENTRY_KEY settings in these files, to the same unique value. sentry_conf.py sentryproj/settings.py Here is an example on how to generate a good unique key that you can use in the files above.\n\u0026gt;\u0026gt;\u0026gt; import base64 \u0026gt;\u0026gt;\u0026gt; import os \u0026gt;\u0026gt;\u0026gt; KEY_LENGTH = 40 \u0026gt;\u0026gt;\u0026gt; base64.b64encode(os.urandom(KEY_LENGTH)) \u0026#39;6+tSEh1qYwDuTaaQRcxUjMDkvlj4z9BU/caCFV5QKtvnH7ZF3i0knA==\u0026#39; Add your email address to SENTRY_ADMINS in sentryproj/settings.py . This will send you emails when an error occurs. SENTRY_ADMINS = (\u0026#39;youremail@example.com\u0026#39;,) Push your code into dotcloud $ dotcloud push sentry . Find out your application url $ dotcloud url sentry Open url in your browser and start using sentry on dotcloud. First things first you should change the admin password from the default one that was created on deployment. The default username and password are found in the mkadmin.py file. Test out sentry using the raven client to make sure it is working as it should. Open up a python shell on your local machine and do the following. Replace the server_url with your sentry url you found out in step 14. Make sure it ends in /store/ . Also make sure you replace my_key with your sentry key\n\u0026gt;\u0026gt;\u0026gt; from raven import Client \u0026gt;\u0026gt;\u0026gt; server_url = \u0026#34;http://sentry-username.dotcloud.com/store/\u0026#34; \u0026gt;\u0026gt;\u0026gt; my_key=\u0026#39;1234-CHANGEME-WITH-YOUR-OWN-KEY-567890\u0026#39; \u0026gt;\u0026gt;\u0026gt; client = Client(servers=[server_url], key=my_key) \u0026gt;\u0026gt;\u0026gt; client.create_from_text(\u0026#39;My event just happened!\u0026#39;) (\u0026#39;48ba88039e0f425399118f82173682dd\u0026#39;, \u0026#39;3313fc5636650cccaee55dfc2f2ee7dd\u0026#39;) If you go to the sentry webpage you should see your test message. If not, double check everything, and see if there was any errors during the send.\nOnce this is all up and running you can install the raven client in your applications, and start sending your logs to sentry.\nOptional: If you don\u0026rsquo;t like the URL they gave you, you can use your custom domain. Assuming your application was sentry.www and your domain was www.example.com you would do the following $ dotcloud alias add sentry.www www.example.com Once you get comfortable with how things work, don\u0026rsquo;t forget to change your DEBUG setting to False. Go ahead and fork my project and get started today.\nFor more info about dotcloud, sentry, and Raven and what you can do with with it. Check out their docs Sentry on DotCloud GitHub repo : https://github.com/kencochrane/sentry-on-dotcloud DotCloud overview: http://docs.dotcloud.com/firststeps/platform-overview/ Sentry Documentation: http://sentry.readthedocs.org/en/latest/index.html Raven Documentation: http://raven.readthedocs.org/en/latest/index.html Links: Virtualenv : http://pypi.python.org/pypi/virtualenv pip : http://www.pip-installer.org/ virtualenvwrapper : http://www.doughellmann.com/projects/virtualenvwrapper/ git : http://git-scm.com/ ","date":"January 28, 2012","externalUrl":null,"permalink":"/blog/2012/01/running-sentry-on-dotcloud/","section":"Posts","summary":"","title":"Running sentry on DotCloud","type":"post"},{"content":"","date":"January 28, 2012","externalUrl":null,"permalink":"/tags/sentry/","section":"Tags","summary":"","title":"Sentry","type":"tags"},{"content":"","date":"January 25, 2012","externalUrl":null,"permalink":"/tags/pci/","section":"Tags","summary":"","title":"Pci","type":"tags"},{"content":"last updated: 12-07-2012 by Ken Cochrane\nUpdate: 12-07-2012 I have added the youtube video and slides from a recent talk I did on Building PCI Complaint Django Applications.\nUpdate: 04-05-2012 This article has been translated into chinese , by Wujun Shen (吴峻申)\nWhen I first started working at CashStar.com three and a half years ago, I had heard about PCI before, but I didn\u0026rsquo;t really know what that meant. Since we were building an ecommerce platform that was going to be accepting credit cards over the internet, I knew we needed to make sure we were fully PCI compliant. We were a startup, we didn\u0026rsquo;t have much money, and any mistake could kill the company. Since I didn\u0026rsquo;t want to be the one to make the mistake, I spent a lot of time doing research on PCI, and what it took to make sure your web application was secure.\nThe first thing that I did was a simple web search, and I was surprised to find out that there really wasn\u0026rsquo;t much information available. Most of the information that was available, wasn\u0026rsquo;t easily understandable, and was a little vague. There were companies that you could hire, that would guide you through the process, but since we didn\u0026rsquo;t have much money, they weren\u0026rsquo;t an option for us. So I did what any geek in my situation would do, I spent a bunch of my time reading and researching as much as I could on PCI, and figured my way through the PCI hell, until we were fully PCI compliant.\nMy goal with this blog post is write all of my information down, so that I can hopefully help others through the process, and also to serve as a reminder to me, so that when I need to do this again in the future, I will remember every last detail. I hoping to keep this as a sort of live document, and I\u0026rsquo;ll try to keep it up to date as time goes forward and things change. If you notice something is incorrect or I\u0026rsquo;m missing something, please leave a comment and I\u0026rsquo;ll do my best to update the post as soon as I can.\nWhat is PCI? # The first thing most people ask is \u0026ldquo;What is PCI?\u0026rdquo;. PCI is short for the Payment Card Industry Security Standards Council. PCI consists of American Express, Discover Financial Services, JCB, MasterCard, and Visa, and was formed on Sept 7th, 2006.\nThe main purpose of creating the PCI SSC, was to come up with a common set of security standards that merchants could use to better protect themselves against hackers. The PCI SSC came up with the Payment Card Industry Data Security Standard (PCI DSS), which consists of 12 requirements, and many sub-requirements that merchants would need to follow in order to accept debit, credit, prepaid, ATM or POS cards from the PCI SSC members.\nWhy was PCI created? # It was created in response to a spike in data security breaches. It gives merchants a guide to help them make sure they are following best security practices when it comes to card holder data. Does PCI affect me? # Do you accept payment online or over the phone with credit/debit cards? Is the credit card information posted to YOUR server? Do you store credit card information, encrypted or not? If you said yes to any of those, then PCI affects you in one way or another.\nPCI DSS Requirements # Here is the list of 12 requirements. As you look them over, you will notice that most of them aren\u0026rsquo;t that complicated, and you might already be doing thing already. Most of them are just common sense, but it is amazing how many people still don\u0026rsquo;t do things, even if it is common sense.\nInstall and maintain a firewall configuration to protect cardholder data Do not use vendor-supplied defaults for system passwords and other security parameters Protect stored cardholder data Encrypt transmission of cardholder data across open, public networks Use and regularly update anti-virus software on all systems commonly affected by malware Develop and maintain secure systems and applications Restrict access to cardholder data by business need-to-know Assign a unique ID to each person with computer access Restrict physical access to cardholder data Track and monitor all access to network resources and cardholder data Regularly test security systems and processes Maintain a policy that addresses information security PCI in Layman\u0026rsquo;s Terms # All Merchants, regardless if credit card data is stored, must achieve and maintain compliance at all times. Merchants cannot store certain credit card information including CVV, track data, magnetic strip or PIN data If you store permitted credit card data, you need to store it in a secure way following the PCI security standards. How does PCI Certification work? # PCI Certification works like this. If you want to accept credit or debit cards you need to agree that you will maintain PCI certification at all times. There are a couple of ways to confirm that you are certified. You need to either fill out a Self-Assessment Questionnaire (SAQ) or a Report on Compliance (RoC). I\u0026rsquo;ll go over the difference in a little bit, but the important part to remember is that you need to fill out some paper work, and then usually submit that paperwork to whomever requests it, usually the company that processes your credit cards and handles your merchant account.\nFill out a Self-Assessment Questionnaire (SAQ) and Find out what level you are Make sure you follow all recommendations for that level Fix any issues Attestation of Compliance (if self assessing) External Auditor (if needed) How to get started? # Identify the individuals that will be responsible for PCI compliance in your organization and assemble a team that includes members from each area. Determine your merchant level (1-4). Determine which SAQ your organization will need to complete. Evaluate whether your organization will try to achieve compliance internally or engage with aQualified Security Assessor (QSA). Engage with an Approved Scanning Vendor (ASV) to start the required external IP vulnerability scans. Make sure that your organization has an Information Security Policy and that it is being enforced. Immediately address any significant deficiencies discovered during the assessment or scan. Retain record of self-assessments, scans, and follow-up activities. Be prepared to provide these documents upon request. What PCI Level am I? # There are 4 PCI compliance level\u0026rsquo;s, and how many transactions you process a year will determine which level you are in.\nMerchant Level # Level 1 : Merchants processing over 6 million Visa transactions annually (all channels) or Global merchants identified as Level 1 by any Visa region\nLevel 2 : Merchants processing 1 million to 6 million Visa transactions annually (all channels)\nLevel 3 : Merchants processing 20,000 to 1 million Visa e-commerce transactions annually\nLevel 4 : Merchants processing less than 20,000 Visa e-commerce transactions annually and all other merchants processing up to 1 million Visa transactions annually\nRequirements # Level 1 : Onsite Security Assessment Required Annually, Network Vulnerability Scan required Quarterly\nLevel 2 : Onsite Security Assessment at Merchants Discretion, Self-Assessment Questionnaire Required Annually, Network Vulnerability Scan Required Quarterly\nLevel 3 : Self-Assessment Questionnaire Required Annually, Network Vulnerability Scan Required Quarterly\nLevel 4 : Self-Assessment Questionnaire Required Annually, Network Vulnerability Scan Required Quarterly\nRoC or SAQ? # If you are a Level 1 then you need to fill out a RoC, if you are level 2, 3 or 4, then you can fill out an SAQ. There are some exceptions to these rules, for example, if you have had a security breach in the past, the credit card companies might require that you complete a RoC even if you aren\u0026rsquo;t a level 1.\nReport on Compliance (RoC) # If you process more than six million credit cards per year (Level 1), you are required to have an on-site PCI assessment and Report on Compliance (RoC) issued by a Qualified Security Assessor (QSA). Other Level 2 organizations may also be required to submit a RoC or choose to do so in anticipation of becoming a Level 1 merchant.\nQSA’s can be engaged to provide this annual review. It includes a review of established processes and procedures for networks, servers and databases in scope for PCI compliance. The engagement involves interviews with stakeholders in your organization, a review of supporting documentation, validation of compliance initiatives and completion of the report itself.\nQSA’s usually encourage their PCI Customers to use a PCI compliance management solution throughout the year. This will assist them with maintaining compliance and should make the on-site engagement and the completion of the RoC faster and smoother.\nSelf-Assement Questionnaire # There are 5 SAQ categories, depending on which category that you fall into, the paper work might be real easy, it might take a lot longer. Here are the 5 categories.\nSAQ-A : Card-not-present (e-commerce or mail/telephone-order) merchants, all cardholder data functions outsourced. This would never apply to face-to-face merchants.\nSAQ-B : Imprint-only merchants with no electronic cardholder data storage, or standalone, dial- out terminal merchants with no electronic cardholder data storage\nSAQ-C-VT : Merchants using only web-based virtual terminals, no electronic cardholder data storage\nSAQ-C : Merchants with payment application systems connected to the Internet, no electronic cardholder data storage\nSAQ-D : All other merchants not included in descriptions for SAQ types A through C above, and all service providers defined by a payment brand as eligible to complete an SAQ.\nSince we are only talking about web applications here, you will most likely only fall into either A, C, or D. Once you know your level you will need to fill out the SAQ for that category. Once you are done you need an Attestation of compliance as well.\nHere is a helpful guide to help you figure out what category you a fall into.\nSAQ-A # There are a lot of different parts to A, but the big one, is that the credit card data never touches your servers. The easiest way to do this is to redirect people to someone else\u0026rsquo;s servers when you want them to enter credit card data. This is common with Paypal, google checkout and Amazon payments.\nAnother way around this is to have your payment page hosted by your credit card gateway. An example of this is authorize.net\u0026rsquo;s Simple Checkout.\nA third way of doing this is what is called \u0026ldquo;transparent redirect\u0026rdquo; or \u0026ldquo;Direct Post\u0026rdquo;, BrainTreePayments was the first to make this popular, but since then Authorize.net has also added it.\nAnd finally the last way, is basically similar to the third way, but it uses javascript to encrypt the credit card data, send it to the credit card processor, and then populate the form with unique tokens, which will be used later on. This approach is used by stripe.\nBrainTree + livingsocial talk about this new approach of end to end encryption of credit card data.\nSAQ-C # If you are hosting the payment form on your own server, and when you hit submit on that form it goes to your server, where you parse the form, get the credit card details out of the fields, build up your request and then send it to the credit card processor yourself. Then you are at least a C. Even if you aren\u0026rsquo;t storing the data, because it is available in your computer memory, and you are touching it with your code, there is risk that something could happen and you would be able to get access to the credit card data.\nSAQ-D # If you don\u0026rsquo;t fall into the other categories then you are a D. SAQ D is sometimes referred to as ROC light, because any organization that has to fill out SAQ D is essentially going through all 12 PCI DSS requirements, albeit on a reduced scale.\nHow much does PCI Cost? # It is really hard to get an accurate value for this because it will be different for everyone, but according to BrainTree here is a chart on how much it costs to become PCI Compliant.\nLevel # of Trans Scope Compliance Audit type\n1 6M+ $125K $586K onsite 2 1M-6M $105K $267K SAQ 3 20K-1M $44K $81K SAQ 4 \u0026lt; 20K ? ? SAQ\nExternal Audits # Level Cost Per Year\nLow End $20K-$30K Average $225K Top 10% $500K+\nIf you are big enough or unlucky enough to require an external audit, it isn\u0026rsquo;t going to be cheap. Audits last a few weeks or more onsite, and cost anywhere from $20K-$30K on the low end. They average around $225K a year, and about 10% of the audits cost over $500K. As you can see this is a really expensive annual cost, and should be avoided if possible.\nIt is also important to point out that this is just the cost of the audit itself, if they find anything wrong in the audit, you will need to pay to fix any of the issues before they will certify you.\nHere are some links where I got my data. http://www.networkworld.com/news/2010/030110-pci-compliance-audit-cost.html http://blog.elementps.com/element_payment_solutions/2009/02/pci-compliance-costs.html https://www.infosecisland.com/blogview/12356-Five-Questions-to-Ask-Your-PCI-Auditor-Before-You-Hire-Them.html PCI 2.0 # On October 26th 2010, PCI DSS version 2.0 was released. Here are some of the highlights.\n132 changes, 2 new ones, the rest are clarifications or additional guidelines Added more guidelines around virtualization, and how it affects PCI Amazon web services (AWS) is now a Level 1 PCI compliant data center. Nitty Gritty # Now that you know what PCI is all about, lets get down to the nitty gritty. The most common questions I\u0026rsquo;m asked is what is the easiest way to become PCI certified. Here is what I tell people.\nFirst off, avoid handling credit card data if you can help it. It has become a lot easier lately with Braintree and stripe. Years ago before these solutions were available, the only way to do it was to use an ugly hosted payment page on your credit card gateway, and it wasn\u0026rsquo;t very good, and hard to integrate, so most people didn\u0026rsquo;t use those solutions.\nNow you have no excuse, let the credit card processor handle all of the credit card data, and it will make your life easier. If you want to see how much easier, go to the PCI security standards website and download the SAQ A and the SAQ C docs. You will notice that the SAQ A is much easier, and a lot less of a hassle.\nAs great as the Briantree and stripe solutions are they can\u0026rsquo;t solve all problems. One common problem is accepting credit card data over an API, more and more common these days with mobile applications. If you can\u0026rsquo;t use one of the other solutions for one reason or another, you can check out Edge Tokenization from Akamai, it will work for both API and web based payment forms. It is pretty expensive, but if you are already using some of akamai\u0026rsquo;s other solutions then this might not be as big of an issue.\nIf you still need/want to accept credit card data on your own server after everything that I said above, then you are going to need to know about some other things. For example, here are a list of common mistakes that most people make.\nCommon PCI Mistakes # Here is a list of common mistakes most people make. I\u0026rsquo;m listing them here so that you can catch these mistakes before it is too late. If I missed any, let me know.\nStoring credit card information in plain text # Ideally, you should never store credit card information, but if you have to, you should always encrypt the data, so that if someone gets ahold of your data, they won\u0026rsquo;t be able to see it unless they put in a lot of effort.\nDefault passwords not changed # I\u0026rsquo;m always surprised to here how weak peoples passwords are, and how most of the time they are still using the first one that was given to them when they started. That is why if you are the one generating a password, make it a secure one, so that if the people don\u0026rsquo;t change the password like you told them too, it will at least be a secure one to begin with.\nThere are really good password management tools on the market today, I recommend using one of them. One of my favorites is 1password.\nRemove all programs not needed from your servers # There are a couple reasons why you would want to remove any programs/software from your computer if you are not using it. The first one, it will take up less space, and if it isn\u0026rsquo;t running it will free up processor and RAM, a faster system is always good. The second reason is so that you don\u0026rsquo;t have to maintain the security patches for something you aren\u0026rsquo;t using. So, the first step you should do when you bring a new server online is to remove all of the stuff you aren\u0026rsquo;t using. You can always add it back later.\nUse a Firewall # You should always use a firewall, it doesn\u0026rsquo;t matter if it is a hardware firewall or a software firewall, use it, and never turn it off. In some of my production systems I run both a hardware firewall coming into my private network and then a software firewall on each system. Some people think this is overkill, but I would rather be safer then sorry.\nJust running the firewall is only part of it, you need to know how the firewall is setup, and why. You should always have a document around with a list of which ports are open and why. This will be very helpful later on, when you get audited and they want to know what ports are open, and the reasons for it.\nYou should do a quarterly review of your firewalls to make sure they match your documentation, and to see if any of the ports that were previously open still need to be open. Systems change over time, and sometimes you will remove a service that isn\u0026rsquo;t needed anymore, and when that happens you should also block the port.\nYou could also use a service like CloudFlare that protect your website from a range of online threats from spammers to SQL injection to DDOS. It is easy to setup, and your code changes should be minimal at most.\nPoorly coded websites # If the programmers who are writing your web application aren\u0026rsquo;t careful, and don\u0026rsquo;t know what they are doing, they could write bad code which could result in SQL injection and other vulnerabilities.\nCross Site Scripting (XSS) is becoming a more and more common way of attacking websites these days, so make sure you are careful of that as well.\nMake sure you always conduct code reviews, and use application penetration testing before you put your code into production.\nLack of monitoring and logging # It is amazing how many companies have no system or application monitoring, it is like they are running blind, they have no idea when something is going wrong until their customers tell them. You should have as much monitoring and logging as possible, so that you know what is happening with your system at all times. If you don\u0026rsquo;t log when things are going well, then when stuff starts going bad you will have no idea what things are suppose to look like when things are good.\nHere is a list of tools that will help you with your logging and monitoring.\nPingdom Is a website monitoring tool, they will tell you when your site is down. Nagios offers complete monitoring and alerting for servers, switches, applications, and services. Cacti is a complete network graphing solution designed to harness the power of RRDTool\u0026rsquo;s data storage and graphing functionality. Sentry Open Source realtime event logging and aggregation platform Loggly Log management cloud service for centralized log search and analysis, time series data. graphite Scalable Realtime Graphing server collectd is a daemon which collects system performance statistics periodically and provides mechanisms to store the values in a variety of ways, for example in RRD files. monit Easy, proactive monitoring of Linux/Unix systems, network and cloud services. munin Munin is a networked resource monitoring tool that can help analyze resource trends. New Relic is the only tool you need to pinpoint and solve performance issues in your Ruby, Java, .NET, PHP and Python apps. Pager Duty Phone \u0026amp; SMS alerting and on-call scheduling for Nagios, Zenoss, Munin, Monit, and most other IT monitoring tools. Missing security patches # It is important that you regularly schedule applying all security patches on all of your systems. This is a no brainer but it is amazing how much this doesn\u0026rsquo;t happen.\nYou should also subscribe to all of the security alert email lists for any of the products that you are using, as well as paying attention the following list of websites below. The sooner you get notified of a potential problem the sooner you can fix it before it effects you.\nhttp://www.us-cert.gov/cas/ http://seclists.org/ http://www.sans.org/newsletters/ Not using SSL for payment page # Another no brainer, but sometimes it happens. You should add code to your web applications that check to make sure that the payment pages are served over SSL, if not, do a redirect to the SSL version of the page.\nAn easy way to do this is to serve the whole site over SSL all of the time, and then do a simple redirect with your web server from port 80 (http) over to port 443(https). This will guarantee that all traffic is served over SSL all of the time.\nLogging payment information # One of the most common mistakes that I see is when someone has their logging setup to print out data from the payment form to the logs. This is great for debugging purposes but bad for PCI. You should always strip out the important information out of the request before logging it. You can replace the credit card number with **last4 and get the same result.\nAnother common mistake that is similar is dumping all of the data when there is an error and emailing it to the developers. If you do this as well, make sure you strip out the credit card info first or else that person\u0026rsquo;s credit card information is now emailed all over the place, which isn\u0026rsquo;t good at all.\nCredit card data that can be stored # It is important that you NEVER EVER store credit card information in the database, even if it is encrypted. It isn\u0026rsquo;t worth the hassle, risk and the cost of handling an external audit. But if you absolutely insist, here is something you need to know.\nIf for some reason you ignore my advice and decide to store credit card data anyway, here is a little chart that will show you which data is allowed to be stored, and If it needs to be encrypted or not.\nAccording to 3.3 Mask PAN when displayed (the first six and last four digits are the maximum number of digits to be displayed). That means, you need to do something like this *****1234 Visa instead of the actual credit card number. This is pretty common these days.\nAccording to 3.4 : Render PAN unreadable anywhere it is stored (including on portable digital media, backup media, and in logs) by using any of the following approaches:\nOne-way hashes based on strong cryptography (hash must be of the entire PAN) [ One-way hash functions such as the Secure Hash Algorithm (SHA) based on strong cryptography can be used to render cardholder data unreadable. Hash functions are appropriate when there is no need to retrieve the original number (one-way hashes are irreversible) To complicate the creation of rainbow tables it is recommended, but not a requirement, that a salt value be input to the hash function in addition to the PAN.] Truncation (hashing cannot be used to replace the truncated segment of PAN) Index tokens and pads (pads must be securely stored) Strong cryptography with associated key-management processes and procedures +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+ | | Storage Permitted | Protection Required | +=====================+======================+==========================+ | | Cardholder Data | | +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+ | Account Number | \u0026gt; Y | \u0026gt; Y | +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+ | Cardholder data | \u0026gt; Y | \u0026gt; N | +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+ | Expiration Date | \u0026gt; Y | \u0026gt; N | +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+ | Service Code | \u0026gt; Y | \u0026gt; N | +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+ | | **Authentication | ta** | | | Da | | +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+ | Magnetic Strip | \u0026gt; N | \u0026gt; n/a | +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+ | CVV | \u0026gt; N | \u0026gt; n/a | +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+ | Pin Data | \u0026gt; N | \u0026gt; n/a | +\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;-+\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026mdash;\u0026ndash;+\nTokenization # If you need to store credit card information, it is best to use a tokenization service instead of storing it yourself. You store the credit card information in their system. They give you a unique token that you use for all future transactions against that credit card. These types of service are pretty common these days, just ask your credit card processor if they have such a service. Here are a couple of credit card processors that provide this sort of service.\nAuthorize.net CIM BrainTree Vault Stripe.com Recurly.com subscription based billing Data Centers # When you are dealing with PCI compliance you need to worry about the full stack, not just your application, but also the server the application lives on, the network your server is connected to, and the data center your server lives in. The first thing you will want to do is contact your your hosting provider to see if they are PCI compliant, and if so, you might want to request a copy of their PCI documents for your records (you might need them later). Usually hosting providers that are PCI compliant brag about it on their web pages, so that is usually a good place to start.\nThe smaller the hosting company that you deal with the smaller the chance you will be PCI compliant. If you are just using a shared hosting plan, and paying $20/month, most likely you are not compliant. You might get lucky, but I doubt it. If you are using a PAAS or a cloud provider, you will also most likely be out of luck.\nHosting in the Cloud # Amazon Web Services (AWS) has recently had their data centers meet PCI compliance, but what is important to note is that just because the data center in compliant, doesn\u0026rsquo;t mean that your application is going to be. If you put your application on EC2, and you accept credit card data that is getting processed on those EC2 instances, you will need to make sure that you also have an Intrusion Detection System (IDS) amongst other things in place or else you aren\u0026rsquo;t PCI compliant. All of the good IDS\u0026rsquo;s are hardware based, and have someone monitoring the traffic at all times. You can\u0026rsquo;t install those systems in AWS, so you will need to rely on a software based solution, which isn\u0026rsquo;t as good, and adds another layer of complexity to your network stack.\nRackSpace offers a hybrid cloud hosting setup, which allows you to have hardware firewall, IDS, Load balancers, cloud web servers and hardware database servers. But even in this setup, it isn\u0026rsquo;t PCI compliant, at least I haven\u0026rsquo;t been able to get RackSpace to tell me it is yet.\nThere are other cloud providers that might be able to offer you a complete PCI compliant solution, but I\u0026rsquo;m guessing they are going to cost more money. If you know of one, please let me know and I\u0026rsquo;ll update this. Terremark might have something.\nSecurity Scanners # A key part of the PCI certification is the 3rd party security scanning requirement. Basically you have to pay one of the certified and approved security scanning companies to scan you network, server, application every so often, and if it finds any issues, you will need to fix those, and scan again until you pass their tests. Once you pass the scans they will give you a certificate that you can attach to the rest of your PCI documentation.\nI have used a company called ControlScan in the past, and I have also used Qualys, but I\u0026rsquo;m sure there are a ton of others out there. Pick the one that looks the best for you. Here is a link to a list of PCI approved scanning vendors\nIntrusion Detection Systems # Intrusion Detection Systems (IDS) basically sit in front of your network and watch all of the network traffic coming into your network. It looks to see if it notices anything out of the ordinary, of if people are trying to use known attacks, and if it finds something it will let you know. They have hard ware and software based solutions. They range in price from free to thousands of dollars a month. They all have different features and abilities, it is best to pick one that has what you need, that you are comfortable maintaining.\nI have used AlertLogic\u0026rsquo;s hardware based IDS, and it works well. They have a pool of on call people who monitor the devices and if something gets triggered they look it over, and act accordingly.\nHashing credit card numbers # Here is a great example on why hashing credit card numbers isn\u0026rsquo;t a good idea. I\u0026rsquo;m borrowing some of this from these two links.\nhttp://en.oreilly.com/rails2011/public/schedule/detail/19466 http://www.integrigy.com/security-resources/whitepapers/Integrigy_Hashing_Credit_Card_Numbers_Unsafe_Practices.pdf Just because you are following PCI rules doesn’t mean you are invincible, you still have to use your common sense.\nPCI DSS section 3.4 [pdf]: Render PAN, at minimum, unreadable anywhere it is stored .. by using any of the following approaches: Strong one-way hash functions (hashed indexes)\nVerify that data is rendered unreadable using one of the following methods: one-way hashes (hashed indexes) such as SHA-1\nBasically what this is saying is that you are allowed to store the first 6 digits of a credit card (BIN) as well as the last 4 digits of the credit card. Credit cards are between 13-16 digits in length and the last digit is the check digit (Luhn algorithm).\nLet\u0026rsquo;s see how hard it would be to figure out this credit card number. 4012888888881881\nIf we start with a full 16 digits that means that we have 10^16 or 10,000,000,000,000,000 (10 Quadrillion) Possible Card Numbers, if we didn\u0026rsquo;t know anything about the card.\nSince we are storing the credit card type, we know this is a visa, visa credit cards all start with a 4 so that means that is could be 4XXXXXXXXXXXXXXX or 4,000,000,000,000,000 (4 Quadrillion) Possible Card Numbers, we just cut the number of possible cards in more then half.\nIf we also store the bin (first 6 digits) and the last 4 digits, then it would look like this. 401288****1881 or 1,000,000 (1 million) possible card numbers.\nStarting with that lets try to write a simple cracker (Ruby)\nhashed_card_number = \u0026#39;62163a017b168ad4a229c64ae1bed6ffd5e8fb2d\u0026#39; masked_card_number = \u0026#39;401288******1881\u0026#39; Code\nrequire \u0026#39;digest/sha1\u0026#39; def reverse_hashed_card_number( hashed_card_number, first_six, last_four) 0.upto(999_999) do |i| card_number_to_test = \u0026#34;#{first_six}%06d#{last_four}\u0026#34; % i hashed_to_test = Digest::SHA1.hexdigest(card_number_to_test) if hashed_card_number == hashed_card_number_to_test return card_number_to_test end end end Let\u0026rsquo;s run it\nBenchmark.measure do puts reverse_hashed_card_number( \u0026#39;62163a017b168ad4a229c64ae1bed6ffd5e8fb2d\u0026#39;, \u0026#39;401288\u0026#39;, \u0026#39;1881\u0026#39; ) end.real 4012888888881881 =\u0026gt; 5.33522081375122 In 5.3 seconds it was able to crack the hash, if you use only a SHA-1 hash. We could possibly make it even faster if we did a luhn check on the number before we ran the hash, and if the luhn check fails then we know the number isn\u0026rsquo;t valid and there is no need to run the hash. Since the hash function is going to be slower then the luhn check it should speed things up.\nRainbow Tables + Salts # Since we know that there is a finite number of credit cards, we could pre-calculate the hash code for every single one of the 10 Quadrillion possible card values, and store those in a lookup table. Then when ever I wanted to crack a credit card hash, all i would need is the credit card hash, and I would be able to figure out the value of that card, very quickly. Storing all of the known values in a table like this is called a Rainbow table.\nIdeally if you are going to hash a credit card, don\u0026rsquo;t use SHA-1, or MD5, use one of the newer SHA versions, SHA-256 or above, and also use a salt. A salt is basically a second unique value that you always use when hashing, to generate a different salt then you would normally get with just the credit card number.\nSince I won\u0026rsquo;t have your salt when I generate my rainbow table, my rainbow table will be no good. It adds yet another layer of security. Make sure you don\u0026rsquo;t lose your SALT or else you will have to start over from scratch. Treat your salt like a password, and keep it safe.\nDo I really have to worry about being hacked? # Here is a short list of companies that have been hacked recently. If they can get hacked, so could you.\nTJ Maxx Bank of America Citigroup BJ\u0026rsquo;s wholesale club Hotels.com LexisNexis Polo Ralph Lauren Wachovoa Heartland Payment Systems Hannaford What could happen if you were Hacked? # Banned from accepting credit cards Loss of reputation and customers Fines up to $500,000 per incident Litigation (you could be sued) What if I was breached? # In the event of a security incident, merchants must take immediate action to:\nContain and limit the exposure. Conduct a thorough investigation of the suspected or confirmed loss or theft of account information within 24 hours of the compromise Alert all necessary parties. Be sure to notify: * Merchant Account Provider * Visa Fraud Control Group at (650) 432-2978 * Local FBI Office * U.S. Secret Service (if Visa payment data is compromised) Provide the compromised Visa accounts to Visa Fraud Control Group within 24 hours. Within four business days of the reported compromise, provide Visa with an incident report. Build PCI Complaint Django Applications # I recently gave a talk on Build PCI Complaint Django Applications, at DjangoCon US 2012 in Washington D.C. Here are my slides and the video of my talk.\nSlides # Video # Links: # Akamai edge tokenization PCI Security Standards American Express PCI pages Discover Financial Services PCI pages JCB International PCI pages MasterCard Worldwide PCI pages Visa Inc PCI pages Visa Europe PCI pages ","date":"January 25, 2012","externalUrl":null,"permalink":"/blog/2012/01/developers-guide-to-pci-compliant-web-applications/","section":"Posts","summary":"","title":"The Developers Guide to PCI Compliant Web applications","type":"post"},{"content":"","date":"January 15, 2012","externalUrl":null,"permalink":"/tags/djangocms/","section":"Tags","summary":"","title":"Djangocms","type":"tags"},{"content":"ActiveState has recently started their own PAAS based on CloudFoundry this blog post is to help you get up and running quickly with a Django CMS installation, and hopefully give you enough information to get your own applications on there as well.\nTo keep things simple, I\u0026rsquo;ll just go through the steps that are required in order to get this up and running and add some comments along the way. If I missed anything or if you have any questions, please ask in the comments.\nRunning on Stackato # Register for an account Go to http://community.activestate.com/stackato and register for your account. They are currently in beta, and you need to be approved before they will give you access.\nInstall the client There are a few different options, follow the steps outlined here. http://docs.stackato.com/quick-start.html#stackato-client-setup\nI tried the option where you download the file, and rename it and put it in your path. It wasn\u0026rsquo;t complicated, but it wasn\u0026rsquo;t clean either. I chose this option because I didn\u0026rsquo;t want to install pypm, so I\u0026rsquo;m not sure if that way is easier or not.\nRegister your client to the cloud Now that you have the client installed you need to tell it which cloud to deploy too. With Stackato, you can run your own local cloud as well as using the sandbox that ActiveState provided. To make things simple, I\u0026rsquo;m going to use the sandbox. If you want to use the local cloud you will need to read the directions on how to use the local cloud option.\n$ stackato target api.sandbox.activestate.com \u0026gt; Successfully targeted to [https://api.sandbox.activestate.com] Login to stackato Once you point your client in the correct location, you will need to login to the cloud. You can find your login information on your activestate account page. Type the following and answer the questions when prompted.\n$ stackato login \u0026gt; Successfully logged into [https://api.sandbox.activestate.com] Download this github repo To make things easier, I have made a simple django project that has all of the configuration information setup so that you don\u0026rsquo;t need to do anything if you want to use djangoCMS. Look at my github repo, and see how I did things if you want to get your own application up and running. If you want to use djangoCMS, then all you have to do is follow these steps.\n$ cd ~/projects $ git clone git://github.com/kencochrane/django-cms-stackato.git $ cd django-cms-stackato Deploy the project to stackato Once you have your application ready, you can push the application to the cloud. When you do this it will prompt you for a bunch of questions, answer them and keep track of what you picked for a website url, because you will need that later. My application is called myblog, but you can put whatever you want, just change myblog with your name.\n$ stackato push myblog Initialize the database (optional) I have set this up so that it should happen automatically at deployment see the stackato.yml file for more details. If you want to run the commands outside of deployments this is what you can do.\n$ stackato run myblog python mycms/manage.py syncdb --noinput Run south migrations (optional) I have set this up so that it should happen automatically at deployment see the stackato.yml file for more details. If you want to run the commands outside of deployments this is what you can do. It is important to note I had to run more then once since it was killed the first time. Maybe it took too long?.\n$ stackato run myblog python mycms/manage.py migrate --noinput Collect the static files (optional) I have set this up so that it should happen automatically at deployment see the stackato.yml file for more details. If you want to run the commands outside of deployments this is what you can do.\n$ stackato run myblog python mycms/manage.py collectstatic --noinput Create the django admin account Now that you have your application installed and you have your database setup, you need to create the django admin, you can do that with ths django management command. Make sure you replace the variables with your values.\n$ stackato run myblog python mycms/manage.py createsuperuser --username=admin --email=admin@example.com --noinput Change the password for the admin user You need to set a password for the admin account so that you can login. Pick a more secure password then the example I have here. (notice it is changepassword2 not changepassword)\n$ stackato run myblog python mycms/manage.py changepassword2 admin secret123P@ssw0rd! Open up the url in your browser When you open up the URL that you picked when you deployed in your browser you should find the DjangoCMS pony welcome page. If not, try debugging using some of the tips below.\nConclusion # That is it, I did all the hard work, so you should be able to follow those simple steps and get djangoCMS up and running in no time. Once you get that working, play around with it, and let me know what you think. Have you tried the other PAAS options yet, if not check those out as well, and then let me know which ones you like better and why. I have written blog posts about most of them at this point, so feel free to check those out (links below), and have fun playing around.\nOther Useful Information # Starting an application if it isn\u0026rsquo;t running # $ stackato start myblog Restarting an application # $ stackato restart myblog Stopping an application # $ stackato stop myblog Updating application after it is already deployed # $ stackato update myblog Find out how many instances you have running # $ stackato stats myblog Find out which apps you have installed, and their status # $ stackato apps Find out what logs you have for your applications # $ stackato files myblog logs Viewing logs for your app # $ stackato logs myblog --all Running cat on a particular log file # $ stackato run myblog cat ../logs/myapp-err.log Links # My github repo for this blog post: https://github.com/kencochrane/django-cms-stackato Stackato Client command reference: http://docs.stackato.com/commands.html#command-ref-client stackato.yml reference: http://docs.stackato.com/client.html#configure-stackato-yml Stackato quick start guide: http://docs.stackato.com/quick-start.html Stackato Sandbox Ground Rules, Content Policy and Quotas: http://docs.stackato.com/sandbox.html ActiveState Account page: https://account.activestate.com/ pip : http://www.pip-installer.org/ git : http://git-scm.com/ My other articles related to PAAS: # My Experiences with ep.io AppHosted.com Django Hosting Service Review My Day in Gondor.io Deploying my Django application to DotCloud.com DjangoZoom.com Review Django hosting roundup Installing DjangoCMS on Heroku in 13 easy steps Installing DjangoCMS on dotCloud in 12 easy steps Developers guide to Running Django Applications on Heroku Installing a Django application on Red Hat\u0026rsquo;s OpenShift PAAS Update # 2/16/2012: Full disclosure. On Feb 16th 2012, I accepted a job with dotCloud a competitor to Stackato. I plan on keeping this blog post up to date and impartial. If you think there are any errors, please let me know in the comments.\n","date":"January 15, 2012","externalUrl":null,"permalink":"/blog/2012/01/getting-djangocms-up-and-running-on-stackato/","section":"Posts","summary":"","title":"Getting DjangoCMS up and running on ActiveState's Stackato","type":"post"},{"content":"","date":"January 15, 2012","externalUrl":null,"permalink":"/tags/stackato/","section":"Tags","summary":"","title":"Stackato","type":"tags"},{"content":"It seems like everyone has their own PAAS solution these days, and if they don\u0026rsquo;t have one yet, it is just a matter of time before they will. Red Hat has recently joined in on the fun with their OpenShift platform.\nI decided to take it for a test drive, and share my results with you. This service is still in beta and things are changing all of the time, so these notes might not work in the future, take that into consideration when using it as a guide.\nThere really isn\u0026rsquo;t much python documentation for this platform, and what documentation there is, is either a little out of date, or missing some important steps. Hopefully this guide will help you get your application up and running.\nOpenShift is divided into two parts, Flex and Express.\nFlex: # \u0026ldquo;Dedicated cloud solution. Get more control over your apps, or move your existing applications to the cloud with ease! Flex is a dedicated cloud solution that provides everything you need to easily scale, provision, deploy, and monitor your applications.\u0026rdquo;\nFlex runs on top of your amazon EC2 account, and currently only supports Java and PHP. It is targeting the enterprise crowd and has more features compared to Express.\nExpress: # \u0026ldquo;Shared cloud solution. Express is a free, cloud-based application platform for Java, Perl, PHP, Python, and Ruby applications. It\u0026rsquo;s super-simple—your development environment is also your deployment environment: git push, and you\u0026rsquo;re in the cloud!\u0026rdquo;\nExpress runs on Red Hat\u0026rsquo;s servers, and currently supports Java, Ruby, PHP, Perl and Python apps. It doesn\u0026rsquo;t have the same features of Flex, but if you don\u0026rsquo;t need those advanced features, then express might be fine for you. Since I\u0026rsquo;m interested in deploying python apps, and Express is the only one that supports python apps, that is the one I\u0026rsquo;ll show you today.\nTo make things easier to show you how things work, I\u0026rsquo;ll create a simple django application and walk you through the steps to deploying it. I\u0026rsquo;m going to deploy a djangoCMS to express that will connect to a mysql database.\nSteps: # 1. Create an account # Create an account by filling out the form at http://openshift.redhat.com/ and don\u0026rsquo;t forget your username and password, you will need that later.\n2. Install Perquisites: # git python openshift client **Git:**\nI\u0026rsquo;m not going to go over the steps to install git, for more info, you can get it from the git website. http://git-scm.com/download\nPython:\nIf you are using a Mac or linux, it most likely already has python installed, if you are using windows, you you don\u0026rsquo;t have python installed. Goto the python website and you will find instructions on how to install python on your system. http://python.org/download/\nOpenShift client:\nFor instructions on how to Install the client go to this page. https://openshift.redhat.com/app/express#mac Here are my steps for Mac OSX\n$ sudo gem install json_pure $ sudo gem install rhc 3. Create a domain. # Now that you have everything installed, your first step is to create a domain. Run the following command and replace $mydoman and $loginemail with your own domain and the login email you used when creating an account.\n$ rhc-create-domain -n $mydomain -l $loginemail 4. Create an application # Now that you have a domain, you need to create an application that you can deploy to that domain. Running the following command will create a new wsgi application called \u0026lsquo;blog\u0026rsquo;. You can name your application anything you want, within reason. We pick wsgi, because our python application will be wsgi compatible.\n$ rhc-create-app -a blog -t wsgi-3.2 5. Add mysql to your app # We want to use mysql as our database backend for this blog, so we need to add mysql to our application using the command below.\n$ rhc-ctl-app -a blog -e add-mysql-5.1 6. Add phpmyadmin to help you manage your database # OpenShift doesn\u0026rsquo;t give you direct access to your database, so you will need a way to manage your database a different way. They provide the ability to add PHPMyAdmin to your app so that you can get to your data. This step is optional, but recommended.\n$ rhc-ctl-app -a blog -e add-phpmyadmin-3.4 7. Add this upstream repo from github # When you create your application, it will create a directory with a bunch of other files and directories in it. If you were starting from scratch and building up your application you would start from here. To make things easier I created a project on github that will allow you to get up and running with DjangoCMS much faster. In order to use my project you will need to run the following commands so that it will pull down the code into your project.\n$ cd blog $ git remote add upstream -m master git://github.com/kencochrane/django-cms-openshift.git $ git pull -s recursive -X theirs upstream master Once the code is downloaded, you can take a look at the code, change anything you want and when you are ready you can deploy the app.\n8. Deploying the app # To deploy all you need to do is push the repo upstream into open shift. To this, you just need to run the following command.\n$ git push When it pushes you application into open shift it will do the following. (without jenkins add-on installed)\nstop app *pre_build *build start_dbs *deploy start_app *post_deploy The steps marked with a * are scripts that are located in .openshift/action_hooks/\u0026lt;script_name\u0026gt; if you have something in those files it will run them, and if not, it will pass right by. These are very helpful if you want something to happen at points of the deployment process. For example, in the deploy script you will notice that I have some code to kick off the django syncdb, migrate, and collect static commands. These will run every time I deploy the app.\nI have also created a django management command that will check to see if there is a django admin account created and if not, it will create one and set the default password. I had to do this because there is currently no way that I know of where you can kick off django management commands after the deployment is finished. It will only create the admin account once, and every other time it will just get ignored.\nIMPORTANT: It is important to note that if you want to do anything related to the database, you can\u0026rsquo;t do it in the pre_build or build scripts, because the database isn\u0026rsquo;t available yet. This one thing caused me lots of pain, because I couldn\u0026rsquo;t figure out by my migrations were not working. If you do make the mistake of trying to do something database related in the build script you will see an error like this.\n\u0026ldquo;remote: ERROR 2003 (HY000): Can\u0026rsquo;t connect to MySQL server on \u0026lsquo;xxx.x.xx.x\u0026rsquo; (111)\u0026rdquo;\nThat\u0026rsquo;s it, you can now checkout your application at (default admin account is admin/\u0026lt;password given at deploy time\u0026gt;):\nhttp://blog-$yourdomain.rhcloud.com\nOnce you visit the page you should see the djangoCMS default page. First things first, login to the django admin, and change the password from the default password to something secure. Then get started building your own app.\nHelpful tips: # Viewing logs: # If you would like to view your logs to see what is going on with your application you just need to run this command.\n$ rhc-tail-files -a blog Application information: # If you would like more information about your application you can run this command.\n$ rhc-user-info Also checkout the README file they add when you create an application, it is pretty helpful, and might answer some common questions.\nApplication dependencies: # OpenShift uses virtualenv but it doesn\u0026rsquo;t use pip, it depends on the dependencies be listed in the setup.py file. During the deploy process I did notice that it installed pip, so it might be possible to add a requirements.txt file, and then in your build action_hook script call pip install -r \u0026lt;path\u0026gt;/requirements.txt but I\u0026rsquo;m not sure if this is supported, or if it will cause problems, so it might be best to stick with what they have for now.\nStatic media: # If you look in wsgi/static/.htaccess there a rewrite rule to get the media to work correctly, you can use this trick for other apache tricks if you want. For more information on this checkout the README file.\nWhat is it open shift running: # Red Hat linux with Apache / mod_wsgi, and mysql 5.1\nWhat type of apps do they support? # Here is the current link which can be found if you run this commands and look at the types.\n$ rhc-create-app -h raw-0.1, php-5.3, jbossas-7.0, rack-1.1, jenkins-1.4, perl-5.10, wsgi-3.2 What else does it support? # Things are changing all of the time, but if you run this command you will get a list of the current supported addons.\n$ rhc-ctl-app -L List of supported embedded cartridges: Obtaining list of cartridges (please excuse the delay)... - metrics-0.1 - mysql-5.1 - jenkins-client-1.4 - 10gen-mms-agent-0.1 - phpmyadmin-3.4 - rockmongo-1.1 - mongodb-2.0 Web based control panel: # They offer a web based control panel to do some of the things you can do with the command line, which will be nice, but it doesn\u0026rsquo;t work right now. It doesn\u0026rsquo;t display the correct information, and it doesn\u0026rsquo;t even show the applications I have created, so I don\u0026rsquo;t trust it. Hopefully these issues will get fixed in the future, and this tool will make it easier for less technical people to get started.\nConclusion: # It is nice to see another platform on the market, it is still pretty rough, and there isn\u0026rsquo;t much documentation, but I found it usable. I\u0026rsquo;m sure once they stabilize things, they will spend more time on the documentation side of things.\nOther Helpful OpenShift links: # My GitHub repo for this article https://github.com/openshift/django-example http://blog.ianweller.org/2011/05/12/openshift-express-first-thoughts/ My other articles related to PAAS: # My Experiences with ep.io AppHosted.com Django Hosting Service Review My Day in Gondor.io Deploying my Django application to DotCloud.com DjangoZoom.com Review Django hosting roundup Installing DjangoCMS on Heroku in 13 easy steps Installing DjangoCMS on dotCloud in 12 easy steps Developers guide to Running Django Applications on Heroku Update # 2/16/2012: Full disclosure. On Feb 16th 2012, I accepted a job with dotCloud a competitor to openShift. I plan on keeping this blog post up to date and impartial. If you think there are any errors, please let me know in the comments.\n","date":"January 14, 2012","externalUrl":null,"permalink":"/blog/2012/01/installing-django-application-on-openshift/","section":"Posts","summary":"","title":"Installing a Django application on Red Hat's OpenShift PAAS","type":"post"},{"content":"","date":"January 14, 2012","externalUrl":null,"permalink":"/tags/openshift/","section":"Tags","summary":"","title":"Openshift","type":"tags"},{"content":"","date":"December 31, 2011","externalUrl":null,"permalink":"/tags/heroku/","section":"Tags","summary":"","title":"Heroku","type":"tags"},{"content":"Do you want to use Django-cms on dotcloud but don\u0026rsquo;t know where to start? All you need to do is follow these 12 easy steps, they will get you on your way.\nCreate a place to store your project\n$ mkdir -p ~/projects\nGo into the projects directory\n$ cd ~/projects\nClone git repo from github, requires git client.\n$ git clone git://github.com/kencochrane/django-cms-dotcloud.git\nGo into the new project directory\n$ cd django-cms-dotcloud\nCreating the virtualenv (using virtualenvwrapper, virtualenv, and pip)\n$ mkvirtualenv \u0026ndash;no-site-packages \u0026ndash;distribute django-cms-dotcloud\nInstalling the dotCloud client http://docs.dotcloud.com/firststeps/install/ (here are the steps for Linux and Mac OSX)\n$ sudo pip install -U dotcloud\nSign up for a dotcloud account https://www.dotcloud.com/accounts/register/ if you haven\u0026rsquo;t already.\nThe first time you use the dotcloud account you will need to add your api key. So type dotcloud and follow the steps. You can find your API key at http://www.dotcloud.com/account/settings\n$ dotcloud\nCreate your dotcloud application\n$ dotcloud create mycmsapp\nPush your code into dotcloud\n$ dotcloud push mycmsapp .\nFind out your application url.\n$ dotcloud url mycmsapp\nOpen url in your browser and start using djangoCMS on dotcloud.\nOptional: If you don\u0026rsquo;t like the URL they gave you, you can use your custom domain. Assuming your application was ramen.www and your domain was www.example.com you would do the following.\n$ dotcloud alias add ramen.www www.example.com\nOnce you get comfortable with how things work, you could add more plug-ins, create your own custom templates and then change your DEBUG setting to False. So go ahead fork my project on github and get started.\nAfter you make changes to your local project directory, you can test it on the server by running the dotcloud push command again.\nFor more info about dotcloud and django-cms and what you can do with with it. check out their docs http://docs.dotcloud.com/firststeps/platform-overview/ https://www.django-cms.org/en/documentation/ Links: dotcloud : http://dotcloud.com Virtualenv : http://pypi.python.org/pypi/virtualenv pip : http://www.pip-installer.org/ virtualenvwrapper : http://www.doughellmann.com/projects/virtualenvwrapper/ git : http://git-scm.com/ ","date":"December 31, 2011","externalUrl":null,"permalink":"/blog/2011/12/installing-djangocms-dotcloud-12-easy-steps/","section":"Posts","summary":"","title":"Installing DjangoCMS on dotCloud in 12 easy steps","type":"post"},{"content":"Do you want to use Django-cms on Heroku but don\u0026rsquo;t know where to start? All you need to do is follow these 13 easy steps, and they will get you on your way.\nCreate a place to store your project\n$ mkdir -p ~/projects\nGo into the projects directory\n$ cd ~/projects\nClone git repo from github, requires git client.\n$ git clone git://github.com/kencochrane/django-cms-heroku.git\nGo into the new project directory\n$ cd django-cms-heroku\nCreating the virtualenv (using virtualenvwrapper, virtualenv, and pip)\n$ mkvirtualenv \u0026ndash;no-site-packages \u0026ndash;distribute django-cms-heroku\nSign-Up for a Heroku account. https://api.heroku.com/signup\nInstall the Heroku client. http://devcenter.heroku.com/articles/quickstart\nThe first time you use the Heroku client you will need to login using the same information you used when you signed up. Follow the prompts, and it will finish your install.\n$ heroku login\nCreate your heroku application\n$ heroku create \u0026ndash;stack cedar\nPush your code into heroku\n$ git push heroku master\nSync your database and create your admin account.\n$ heroku run python mycms/manage.py syncdb \u0026ndash;all\nRun your database migrations.\n$ heroku run python mycms/manage.py migrate \u0026ndash;fake\nOpen application in your browser and start using djangoCMS on heroku.\n$ heroku open\nOnce you get comfortable with how things work, you could add more plug-ins, create your own custom templates and then change your DEBUG setting to False. So go ahead fork my project on github and get started.\nAfter you make changes to your local project directory, you can test it on the server by running the git push command again.\nFor more info about Heroku and django-cms and what you can do with with it. check out their docs http://devcenter.heroku.com/categories/platform-basics https://www.django-cms.org/en/documentation/ Links: Virtualenv : http://pypi.python.org/pypi/virtualenv pip : http://www.pip-installer.org/ virtualenvwrapper : http://www.doughellmann.com/projects/virtualenvwrapper/ git : http://git-scm.com/ ","date":"December 31, 2011","externalUrl":null,"permalink":"/blog/2011/12/installing-djangocms-on-heroku-in-13-easy-steps/","section":"Posts","summary":"","title":"Installing DjangoCMS on Heroku in 13 easy steps","type":"post"},{"content":"","date":"December 30, 2011","externalUrl":null,"permalink":"/tags/gunicorn/","section":"Tags","summary":"","title":"Gunicorn","type":"tags"},{"content":"","date":"December 30, 2011","externalUrl":null,"permalink":"/tags/newrelic/","section":"Tags","summary":"","title":"Newrelic","type":"tags"},{"content":"Over the past year I have been testing all of the new python platform as a service companies that have popped up, and I have posted my notes on my blog so that everyone can learn from my experiences. ep.io, apphosted.com, gondor.io, dotcloud.com, DjangoZoom.com, Heroku, Django hosting roundup,\nAll and all, the platforms were very similar, they allowed you to easily host your python/django project without having to worry about managing a server or other typical system administration duties. Some of the services were more advanced and had more features then others, but since it was still early in the game, that was expected.\nThe one thing that was different between each service was the way that you have your python project setup. This might not seem like a big deal, but because of the little differences between providers it required me to change my test project every time I wanted to test out a new providers service. These little changes weren\u0026rsquo;t a big deal for me, but it could be a barrier of entry for less technical people. It also makes it harder for someone to change services if they wanted. This probably sounds like a good idea to the platform providers because it makes it a little harder for people to leave their service, but I don\u0026rsquo;t think that was their intention. Looking at the different platforms, and their implementations, you can see they all make sense, and it was just their own way of solving a problem because there was no standard available.\nBefore I started programmed in python, I came from a Java background. In Java they have already solved the deployment problem with their WAR files. For those of you that aren\u0026rsquo;t familiar with Java or WAR files, basically WAR files are just java projects that are laid out in a common directory structure, with a few required configuration files (web.xml, etc) that the application server uses when deploying the application. This directory structure is then zipped up for easy portability. All of the different Java application servers know how to read these files and deploy the applications. It makes installing applications really easy, and if you ever need to change application servers, it shouldn\u0026rsquo;t require any changes at all, assuming you stuck with the standard format. Java WAR files on Wikipedia\nI was lucky enough to attend DjangoCon US 2011 in Portland Oregon this year. At the conference there was a Django Hosting Panel moderated by Sean O\u0026rsquo;Connor where the different hosting providers were asked questions about their platforms. If you didn\u0026rsquo;t get a chance to see it, it is available on blip.tv here. http://blip.tv/djangocon/django-hosting-panel-5572201 Fun Fact: That is me in the front row with the blue shirt and green hat.\nDuring the panel, Sean asked if there was any plans for working on interoperability between the different platforms. Andrew Godwin referenced Java\u0026rsquo;s WAR format and also talked about how he has already started to talk to others about this idea at DjangoCon EU and would like to see it move forward. The others in the panel seemed to agree that something could be done, but it didn\u0026rsquo;t seem like it was urgent to them, or if they really cared/wanted to do it. I can understand their reluctance, because I\u0026rsquo;m sure they have more important things to work on, and this would be very low on their list of things to do. After all whatever comes from this, is going to require changes to their platform in order to support it.\nThat is why I think we (the developer community), need to come up with a standard, propose it to the rest o the python community, with input from the different platform providers, and then leave it up to the different providers to decide if they are going to support it or not. It would be great if everyone supported it, but because this isn\u0026rsquo;t going to make anyone money in the short term, it will be hard to force anyone to do it.\nI\u0026rsquo;m willing to help to create the standard, but where do we start, and whom do we submit the proposal too, the PSF?\nI think the easiest place to start would be to look at the different platforms and find out what they have in common, and go from there. Off the top of my head, I\u0026rsquo;m pretty sure they all support virtualenv and pip for requirements. They are also all wsgi based, but I think that is where the commonality stops.\nFor example they all have a different way for specifying application settings. ep.io has an ini file, dotcloud has a yml file, and gondor has a .gondor/config file. It shouldn\u0026rsquo;t be too hard to come up with a standard format with sane attributes.\nWe could probably all agree on a common project structure, and where to put the common files (requirements.txt for pip requirements and put it in the root of the project, etc.).\nSomething that will be important is to make the standard solid but flexible. Since all of these platforms are a little different we will need to allow some way for them to have a custom section or file that will be platform dependent, so that they can extend the standard without breaking it. These extensions won\u0026rsquo;t be supported by everyone, but it will allow the platform providers the flexibility to move forward without getting held back with a standard process which is normally a slow process.\nThat is also why versioning is important, make sure it is obvious which version of the standard the application is using, so that it doesn\u0026rsquo;t break as changes in the standard are made in the future.\nAs you can see, with a little bit of effort I have a nice start, but where do we go from here? What do you think, is this a good idea? Do you want to help move it forward? Post some comments below, and lets continue this discussion into 2012.\n","date":"December 30, 2011","externalUrl":null,"permalink":"/blog/2011/12/standardizing-python-wsgi-deployment/","section":"Posts","summary":"","title":"Standardizing Python WSGI deployment","type":"post"},{"content":"","date":"December 30, 2011","externalUrl":null,"permalink":"/tags/supervisor/","section":"Tags","summary":"","title":"Supervisor","type":"tags"},{"content":"New Relic recently added support for python to their awesome web application performance tool, and I have been playing with it on a number of projects.\nInstalling and configuring new relic is pretty well covered in their own documentation, so there is no reason for me to repeat that here. One thing that isn\u0026rsquo;t covered in the documentation is how to use new relic if you are using supervisord to control your gunicorn processes, and I\u0026rsquo;ll take this time right now to show you what I did.\nSetting up new relic with supervisord and gunicorn is pretty easy. All that you need to do, is change your supervisor.conf file and then update your supevisor config, and you are good to go.\nHere is the supervisor.conf file for my awesome app, before I installed new relic.\nNote: These are not my real conf files, they have been changed to protect the guilty, so please excuse any typos.\n[program:awesome_app] directory=/opt/apps/awesome_home/awesome_app/ command=/opt/apps/awesome_home/bin/python2.6 /opt/apps/awesome_home/bin/gunicorn_django -c /opt/apps/awesome_home/awesome_app/conf/gunicorn.conf user=aweman autostart=true autorestart=true environment=HOME=\u0026#39;/opt/apps/awesome_home/awesome_app/\u0026#39;,DJANGO_SETTINGS_MODULE=\u0026#39;settings\u0026#39; After I installed new relic. All you need to do is add the \u0026rsquo;newrelic-admin run-program\u0026rsquo; command before the \u0026lsquo;gunicorn_django\u0026rsquo; command and add an ENV variable called NEW_RELIC_CONFIG_FILE that is pointing to your newrelic.ini file.\n[program:awesome_app] directory=/opt/apps/awesome_home/awesome_app/ command=/opt/apps/awesome_home/bin/newrelic-admin run-program /opt/apps/awesome_home/bin/gunicorn_django -c /opt/apps/awesome_home/awesome_app/conf/gunicorn.conf user=aweman autostart=true autorestart=true environment=HOME=\u0026#39;/opt/apps/awesome_home/awesome_app/\u0026#39;,DJANGO_SETTINGS_MODULE=\u0026#39;settings\u0026#39;,NEW_RELIC_CONFIG_FILE=/opt/apps/awesome_home/awesome_app/conf/newrelic.ini Now that you have the new configuration setup, you will need to let supervisord know that you have changed t he configuration for that app. If you run the update command it will prompt supervisord to reread the configuration file for that app, and reload the config, and then restart the application with the new configuration.\n$ supervisorctl update Another thing that is important to note here, is the fact that New Relic currently doesn\u0026rsquo;t work well with Gunicorn in gevent mode. If you try to use gevent with gunicorn and new relic, it may not start up at all, or just not work as it should. Here is what they say in the Known Issues section of their docs.\nGunicorn gevent mode - When using gevent mode of gunicorn and the \u0026rsquo;newrelic-admin run-program\u0026rsquo; command is used to wrap the invocation of gunicorn, the hosted web application can fail in strange ways. One way this is manifesting is with requests blocking for a period of 1 minute. The cause of the problem is believed in this case to specifically relate to the order in which module imports are occuring. The monkey patching performed by gevent is not working properly for the case where the Python threading module is imported before the gevent monkey patching routine is run.\nBecause of this, I have changed my gunicorn\u0026rsquo;s to use eventlet when using new relic, and that seems to work fine. I normally prefer to use gevent, so hopefully they will be able to fix the issue with gevent so I can revert back to that setup.\nAll and all I have been pretty happy with new relic, it has helped us find issues with our code that would have been a pain otherwise. There support has been awesome, and they have been adding new fixes/ improvements all the time. Can\u0026rsquo;t wait to see what else they have in store for the future. I would try it out if you can, they have a lite version that is free which even includes server monitoring.\n","date":"December 30, 2011","externalUrl":null,"permalink":"/blog/2011/12/using-new-relic-with-supervisord-and-gunicorn/","section":"Posts","summary":"","title":"Using New Relic with supervisord and gunicorn","type":"post"},{"content":"I was looking at the pycon US 2012 website when I stumbled upon the huge list of sponsors, which is really impressive. It got me thinking. Are all of these companies using python? If so, which ones are the best companies to work for?\nIf it was up to you, and location and money wasn\u0026rsquo;t a factor, what company would you work for and why? If you already work at one of these companies, can you share what it is you use python for, and what it is like working there?\nHere is a list of companies that use Python, in no particular order (most of these are pycon US 2012 sponsors). If I missed a company, please let me know.\nDropbox Heroku Google surveymonkey nebula nasuni microsoft gondor facebook eventbrite new relic zeomega CashStar.com linode dotcloud Docker ccp games revolution systems canonical bit.ly activestate caktus group disqus leapfrog online spotify snoball evite plaidcloud mozilla lab305 walt disney animation studios white oak technologies aldebaran robotics cloud foundry stratasan myyearbook.com threadless cisco kontagent toast driven accense technology net-ng truveris kelly creative tech aarki freshbooks wisertogether bitbucket eucalyptus fwix imaginary landscape cox media group openstack devsar emma shining panda vocollect bigdoor reddit dreamhost Red hat Quora Yelp mixpanel justin.tv YouTube Digg Urban Airship Rackspace Post your answers in the comment section below, and I\u0026rsquo;ll update the post with the most common answers and best responses.\nUpdate 12-30-2011 6:30PM EST: Added Rackspace to the list.\n","date":"December 30, 2011","externalUrl":null,"permalink":"/blog/2011/12/what-are-the-best-python-companies-to-work-for/","section":"Posts","summary":"","title":"What are the best python companies to work for?","type":"post"},{"content":"Heroku the platform as a service (PAAS) company, very popular in the Ruby on Rails community, has now started opening up their platform to languages besides Ruby. They now offer support for Python, Java, Clojure, Node.js, and Scala on their new Cedar stack. Heroku\u0026rsquo;s Cedar stack is still in beta, but that shouldn\u0026rsquo;t stop you from trying it out, and that is just what I\u0026rsquo;m going to do.\n6 months ago I wrote a series of blog posts that reviewed all of the cool new django hosting services that were popping up. My goal was to look at them all, and compare them to find out what they had to offer, and to see if I could use them on my future projects. I\u0026rsquo;m going to review Heroku using the same process that I used to review those other services, so that I can we can compare apples to apples.\nFor those of you who are not familiar with my other blog posts, feel free to check them out now ep.io, apphosted.com, gondor.io, dotcloud.com, DjangoZoom.com, Django hosting roundup.\nFor those of you that are lazy, I\u0026rsquo;ll sum it up pretty quickly. What I\u0026rsquo;m trying to do is pretty simple, I sign up for the service, and just looking at the documentation available, can I get my blog application up and running, and what did I have to do, in order for it to work. I\u0026rsquo;ll describe each step along the way, as a helpful guide for others who might be trying to do the same with their application. If I come across an error, I will tell you what the error was, and how I fixed it, so that if you get the same error, hopefully what I did to fix my problem, will also fix yours.\nHeroku Django Documentation # Before you start using any new platform, you should take the time to read the documentation. I know it doesn\u0026rsquo;t sound like fun,but trust me, it will save you a lot of pain and frustration later on. Here are some links that I found helpful.\nQuickstart guide: http://devcenter.heroku.com/articles/quickstart Python articles: http://devcenter.heroku.com/articles/python Django article: http://devcenter.heroku.com/articles/django Dev Center: http://devcenter.heroku.com/ Prerequisites # Heroku account Heroku command line client Python 2.7, virtualenv, pip, git Must use pip to manage dependencies with a requirements.txt file Signing up for Heroku # Signing up was as simple as can be, all I needed to do was follow this link: https://api.heroku.com/signup and fill out my email address. Go into my email and click on the link they sent, which took me to a page that had me pick my password, and I was done. The whole process took less then 1 minute. Can\u0026rsquo;t beat that.\nVerifying you Heroku account # When you sign up for an account, you have limited access to some features (add-ons, etc.) until you verify your account. Verifing your account is easy all you have to do is enter your credit card information. This proves to them you are real, and also allows you to use some of the paid features. I recommend that you verify your account so that you can get access to some of the nice free add-ons that are available.\nInstalling the heroku command line client # I\u0026rsquo;m using a mac so I\u0026rsquo;m going to install the following version of the heroku command line client (toolbelt). http://toolbelt.herokuapp.com/osx/download\nClick on link, download the file. Find the file that was downloaded (heroku-toolbelt.pkg) and double click it. Follow prompts; enter password when prompted, and click close. Test heroku command line client # Open terminial Type \u0026ldquo;heroku version\u0026rdquo; you should see something like this \u0026ldquo;heroku-gem/2.14.0\u0026rdquo; if so, it worked. if not, you did something wrong. Login to heroku from command line client # Logging into the client will try to find your ssh key, and upload it to heroku for pushing code later on. If you don\u0026rsquo;t have a key it will prompt you to create one. If you have more then one it will ask you which one you want to use.\nNo key found example # $ heroku login Enter your Heroku credentials. Email: adam@example.comPassword: Could not find an existing public key. Would you like to generate one? [Yn] Generating new SSH public key. Uploading ssh public key /Users/adam/.ssh/id_rsa.pub Found more then one key example # $ heroku login Enter your Heroku credentials. Email: \u0026lt;my email\u0026gt; Password: Found the following SSH public keys: 1) apphosted.key.pub 2) id_rsa.pub Which would you like to use with your Heroku account? 2 Uploading ssh public key /Users/ken/.ssh/id_rsa.pub Getting your Django application ready # Now that you have an account and the client installed, you are ready to get started. If you are starting from scratch I recommend following the steps in this tutorial. http://devcenter.heroku.com/articles/django . If you are like me and already have an application that you want to install on to Heroku, keep reading.\nSince I already have a project, I\u0026rsquo;m going to tell you what I needed to do in order to get my blog application up and running on heroku. I followed the instructions from the heroku tutorial and went from there. Heroku depends on git, so I\u0026rsquo;m using a copy of my blog app that is hosted on github: https://github.com/kencochrane/kencochrane_blog_heroku\n$ cd /Users/ken/projects/github $ git clone https://kencochrane@github.com/kencochrane/kencochrane_blog_heroku.git kencochrane_heroku $ cd kencochrane_heroku # making a new virtual environment using virtualenvwrapper. $ mkvirtualenv --no-site-packages --distribute kencochrane_heroku It should automatically activate the environment for you. If it doesn\u0026rsquo;t, run this command.\n$ workon kencochrane_heroku Now that I have my virtualenv, I need to install my requirements using pip.\n$ pip install -r requirements.txt Creating django application on heroku # Creating an application on heroku is easy, you just need to run the following command.\n$ heroku create --stack cedar Creating some-name-4741... done, stack is cedar http://some-name-4741.herokuapp.com/ | git@heroku.com:some-name-4741.git Git remote heroku added Deploying django application to Heroku # Once you are ready you can deploy your django application to Heroku.\n$ git push heroku master If everything went well, it should look something like this.\n$ git push heroku master Counting objects: 209, done. Delta compression using up to 4 threads. Compressing objects: 100% (176/176), done. Writing objects: 100% (209/209), 271.89 KiB, done. Total 209 (delta 28), reused 192 (delta 23) -----\u0026gt; Heroku receiving push -----\u0026gt; Python/Django app detected -----\u0026gt; Preparing virtualenv version 1.6.4 New python executable in ./bin/python Installing setuptools............done. Installing pip...............done. -----\u0026gt; Django settings injection Injecting code into blog/settings.py to read from DATABASE_URL \u0026lt;removed a lot of output that doesn\u0026#39;t need to be shown\u0026gt; Successfully installed South django-activitysync django-debug-toolbar django-disqus django-memcache-status django-tagging django-xmlrpc feedparser httplib2 oauth2 python-memcached python-twitter simplejson yolk docutils Pygments psycopg2 Django Cleaning up... -----\u0026gt; Discovering process types Procfile declares types -\u0026gt; (none) Default types for Python/Django -\u0026gt; web -----\u0026gt; Compiled slug size is 10.3MB -----\u0026gt; Launching... done, v5 http://some-name-4741.herokuapp.com deployed to Heroku Deploying application to Heroku failed, now what? # The very first time you push to master you will get something like this.\n$ git push heroku master The authenticity of host \u0026#39;heroku.com (50.19.85.132)\u0026#39; can\u0026#39;t be established. RSA key fingerprint is 8b:48:5e:67:0e:c9:16:47:32:f2:87:0c:1f:c8:60:ad. Are you sure you want to continue connecting (yes/no)? yes This is normal, just type \u0026lsquo;yes\u0026rsquo;, and you will never be prompted for this again.\nDjango app must be in a package subdirectory # If you django project isn\u0026rsquo;t setup correctly, you will get an error like this.\n$ git push heroku master Counting objects: 199, done. Delta compression using up to 4 threads. Compressing objects: 100% (167/167), done. Writing objects: 100% (199/199), 270.86 KiB, done. Total 199 (delta 26), reused 192 (delta 23) -----\u0026gt; Heroku receiving push -----\u0026gt; Python app detected ! Django app must be in a package subdirectory ! Heroku push rejected, failed to compile Python app To git@heroku.com:some-name-4741.git ! [remote rejected] master -\u0026gt; master (pre-receive hook declined) error: failed to push some refs to \u0026#39;git@heroku.com:some-name-4741.git\u0026#39; This failed, because django app must be in a package subdirectory.. So you need to change the way my django app is setup. Move all of the files under a django project directory except the .git directory and .gitignore and requirements.txt files. For an example of what this looks like, check out the project directory structure on my repo in github.\nHeroku push rejected, no Cedar-supported app detected # Another error you can get is this one. \u0026ldquo;Heroku push rejected, no Cedar-supported app detected\u0026rdquo;\n$ git push heroku master Counting objects: 204, done. Delta compression using up to 4 threads. Compressing objects: 100% (171/171), done. Writing objects: 100% (204/204), 271.34 KiB, done. Total 204 (delta 27), reused 192 (delta 23) -----\u0026gt; Heroku receiving push ! Heroku push rejected, no Cedar-supported app detected To git@heroku.com:cold-summer-4741.git ! [remote rejected] master -\u0026gt; master (pre-receive hook declined) error: failed to push some refs to \u0026#39;git@heroku.com:cold-summer-4741.git\u0026#39; I had this error and I couldn\u0026rsquo;t figure it out. After searching through all of the heroku python docs (all 2 of them), I didn\u0026rsquo;t find anything, so I started doing some trial and error testing, and I finally figure out what was wrong. My django project directory was kencochrane_blog, and this had an underscore, and when I changed it to just kenblog (no more underscore), it worked. I couldn\u0026rsquo;t find anything anywhere that said you can\u0026rsquo;t use underscores for your django app name, so they might want to update their documentation, or possibly fix the bug. I\u0026rsquo;m not sure if this is by design or not, maybe someone can let me know.\nCheck Django application status on heroku # Now that you have pushed your application out onto Heroku how do you know if it is actually running? The Heroku client has a command called ps, which will tell you what your processes are doing.\n$ heroku ps Process State Command ------------ ------------------ ------------------------------ web.1 crashed for 2m python kenblog/manage.py runserver 0... My django application looks like it crashed. Most likely because I didn\u0026rsquo;t setup the database yet, lets look at the logs to see what is causing the problems to be sure.\nChecking Django application logs on Heroku # The Heroku client has a nifty logs command that you can use to view the last chunk of logs for your application. It will combine all of your logs into one view, so that you don\u0026rsquo;t have to have more then one log process running. You can call it directly and get an output similar to the shell command \u0026rsquo;tail\u0026rsquo; and you can all use the \u0026lsquo;\u0026ndash;tail\u0026rsquo; option to follow the logs.\n$ heroku logs --tail 2011-11-27T18:40:00+00:00 heroku[api]: Deploy 5f194d2 by kencochrane@2011-11-27T18:40:00+00:00 heroku[api]: Release v7 created by kencochrane@ 2011-11-27T18:40:00+00:00 heroku[web.1]: State changed from crashed to created 2011-11-27T18:40:00+00:00 heroku[web.1]: State changed from created to starting 2011-11-27T18:40:00+00:00 heroku[slugc]: Slug compilation finished 2011-11-27T18:40:03+00:00 heroku[web.1]: Starting process with command `python kenblog/manage.py runserver 0.0.0.0:12033 --noreload` 2011-11-27T18:40:04+00:00 app[web.1]: Unexpected error: (\u0026lt;type \u0026#39;exceptions.NameError\u0026#39;\u0026gt;, NameError(\u0026#34;name \u0026#39;DATABASES\u0026#39; is not defined\u0026#34;,), \u0026lt;traceback object at 0x1a7b128\u0026gt;) 2011-11-27T18:40:05+00:00 app[web.1]: There is no South database module \u0026#39;south.db.None\u0026#39; for your database. Please either choose a supported database, check for SOUTH_DATABASE_ADAPTER[S] settings, or remove South from INSTALLED_APPS. 2011-11-27T18:40:06+00:00 heroku[web.1]: State changed from starting to crashed Looks like it is an issue with the settings file. If you wanted to take a look at the settings file to see if we can find out what is wrong you can run the following command.\n# (your path will be different) $ heroku run cat kenblog/settings.py \u0026lt;normal settings file stuff with the following added at the end.\u0026gt; import os, sys, urlparse urlparse.uses_netloc.append(\u0026#39;postgres\u0026#39;) urlparse.uses_netloc.append(\u0026#39;mysql\u0026#39;) try: if os.environ.has_key(\u0026#39;DATABASE_URL\u0026#39;): url = urlparse.urlparse(os.environ[\u0026#39;DATABASE_URL\u0026#39;]) DATABASES[\u0026#39;default\u0026#39;] = { \u0026#39;NAME\u0026#39;: url.path[1:], \u0026#39;USER\u0026#39;: url.username, \u0026#39;PASSWORD\u0026#39;: url.password, \u0026#39;HOST\u0026#39;: url.hostname, \u0026#39;PORT\u0026#39;: url.port, } if url.scheme == \u0026#39;postgres\u0026#39;: DATABASES[\u0026#39;default\u0026#39;][\u0026#39;ENGINE\u0026#39;] = \u0026#39;django.db.backends.postgresql_psycopg2\u0026#39; if url.scheme == \u0026#39;mysql\u0026#39;: DATABASES[\u0026#39;default\u0026#39;][\u0026#39;ENGINE\u0026#39;] = \u0026#39;django.db.backends.mysql\u0026#39; except: print \u0026#34;Unexpected error:\u0026#34;, sys.exc_info() This is odd, not sure why this is like this since I\u0026rsquo;m seeing the DATABASES variable above. Let\u0026rsquo;s check to see if the environment variables are there. (the output has been changed to protect the innocent, but you still get the picture). You can also use the command \u0026lsquo;heroku config\u0026rsquo; if you just want to see your application configuration options.\n$ heroku run env Running env attached to terminal... up, run.6 DATABASE_URL=postgres://blah:somethingelse@ec2-555-55-555-555.compute-1.amazonaws.com/morestuff PORT=37265 HOME=/app PYTHONUNBUFFERED=true TERM=xterm COLUMNS=153 PATH=bin:/usr/local/bin:/usr/bin:/bin PWD=/app SHARED_DATABASE_URL=postgres://blah:evenmorestuff@ec2-555-55-555-555.compute-1.amazonaws.com/morestuff LINES=32 This output looks good the DATABASE_URL env variable is there, so that is OK. Looking more closely at my settings.py file I didn\u0026rsquo;t have a DATABASES variable because the last time I used this project, the other hosting provider didn\u0026rsquo;t require that I have one. So I have added a blank DATABASES variable, and that seems to have fixed my issue. Heroku, should make the error message a little nicer, add more notes to their documentation, or even better make their code handle the case when there is no DATABASES variable in the settings file. I have submitted a patch to fix this, so hopefully that will get rolled out in the near future. (https://github.com/heroku/heroku-buildpack-python/pull/8)\nNow if we redeploy the app and look at the logs, they look much better.\n$ heroku logs 2011-11-27T18:59:25+00:00 heroku[api]: Deploy 96be00f by kencochrane@ 2011-11-27T18:59:25+00:00 heroku[api]: Release v8 created by kencochrane@ 2011-11-27T18:59:25+00:00 heroku[web.1]: State changed from crashed to created 2011-11-27T18:59:26+00:00 heroku[web.1]: State changed from created to starting 2011-11-27T18:59:26+00:00 heroku[slugc]: Slug compilation finished 2011-11-27T18:59:31+00:00 app[web.1]: 0 errors found 2011-11-27T18:59:31+00:00 heroku[web.1]: State changed from starting to up Running django management commands on Heroku # Running django management commands are easy you just need to run the following command and replace \u0026lt;command\u0026gt; with the management command you want to run. There are a few examples below.\n$ heroku run python kenblog/manage.py \u0026lt;command\u0026gt; Running django shell on Heroku # If you need to use the interactive python shell with django, you still can, you just need to run the following command.\n$ heroku run python kenblog/manage.py shell Syncing Django Database on Heroku # Now that our application is starting like it should let\u0026rsquo;s sync the db.\n$ heroku run python kenblog/manage.py syncdb Running Django South Migrations on Heroku # $ heroku run python kenblog/manage.py migrate Now lets look at our processes now\n$ heroku ps Process State Command ------------ ------------------ ------------------------------ run.5 complete for 21m cat kenblog/settings.py run.6 complete for 18m env run.7 complete for 1m python kenblog/manage.py syncdb run.8 complete for 1m python kenblog/manage.py migrate web.1 up for 4m python kenblog/manage.py runserver.. Things are looking better, we are up, and it also shows our old commands that we ran.\nif you need to work with the processes you have the following options.\nps:dynos [QTY] # scale to QTY web processes ps:restart [PROCESS] # restart an app process ps:scale PROCESS1=AMOUNT1 ... # scale processes by the given amount ps:stop PROCESS # stop an app process ps:workers [QTY] # scale to QTY background processes Opening your django application in a web browser # If you run this command it will open a web browser and hopefully your site is running as it should\n$ heroku open Running django and gunicorn on heroku # By default heroku deploys with the built in django runserver, which isn\u0026rsquo;t recommended for production. If you are playing around it is ok, but once you get past that, the first thing you should do is switch to something better like guincorn. Switching is quick and painless, and you will be glad that you did.\nAdd gunicorn==0.13.4 to your requirements file. create a new file called Procfile at the same level as your requirements.txt file with the following in it. web: python kenblog/manage.py run_gunicorn -b \u0026#34;0.0.0.0:$PORT\u0026#34; -w 3 Add gunicorn to the installed_apps in settings.py\nthen\n$ git commit -am \u0026#34;use gunicorn\u0026#34; $ git push heroku master Looking at your logs you will see it up and running\n$ heroku logs 2011-11-27T21:06:24+00:00 heroku[slugc]: Slug compilation started 2011-11-27T21:06:32+00:00 heroku[api]: Deploy 7b2eec1 by kencochrane@ 2011-11-27T21:06:32+00:00 heroku[api]: Release v8 created by kencochrane@ 2011-11-27T21:06:32+00:00 heroku[web.1]: State changed from created to starting 2011-11-27T21:06:35+00:00 heroku[web.1]: Starting process with command `python kenblog/manage.py run_gunicorn -b \u0026#34;0.0.0.0:4075\u0026#34; -w 3` 2011-11-27T21:06:36+00:00 app[web.1]: Validating models... 2011-11-27T21:06:37+00:00 heroku[web.1]: State changed from starting to up Running django/celery background tasks on Heroku # Heroku offers you two types of processes, a web process and a worker process. Web processes are used for serving web pages, etc. Worker processes are used for items that run in the background. The heroku tutorial goes over how to setup background task using celery and kombu (http://devcenter.heroku.com/articles/django) if you want more info I would check that out.\nServing Django Static media on Heroku # I\u0026rsquo;m not sure how to serve up static media on heroku, to get my app working I just used the built in \u0026ldquo;django.views.static.serve\u0026rdquo; views for now, this isn\u0026rsquo;t ideal and you would want a more permanent solution. A more long term solution would be to push all static media into Amazon s3 and then use Amazon cloudfront to serve everything.\nUploading files to Heroku with Django # Heroku allows you to upload files to their app\u0026rsquo;s \u0026ldquo;Emphemeral Filesystem\u0026rdquo;: http://devcenter.heroku.com/articles/dyno-isolation#ephemeral_filesystem\n\u0026ldquo;Which the app can use as a temporary scratchpad, but no files it writes are visible to any other \u0026ldquo;Each dyno gets its own ephemeral filesystem, with a fresh copy of the most recently deployed code. During its lifetime the dyno can use the filesystem as a temporary scratchpad, but no files it writes are visible to any other dyno (including other dynos in the application) and any files written will be discarded the moment the dyno is stopped or restarted.\u0026rdquo;\nThis means you can\u0026rsquo;t use this space to store uploaded files, but it should allow you to upload it there for a minute, and then send it over to Amazon S3 for a more permant storage solution. You could use a tool like Django Queued storage for this: https://github.com/jezdez/django-queued-storage\nDatabases # Heroku gives each app a 5MB shared postgresql database for FREE. If you pay $15/month you get upgraded to 20GB of space. They also offer dedicated database instances if you need that. See this page for more details. http://www.heroku.com/pricing#0-0\nThey also have a bunch of other addons (redis, mongodb, couchdb, amazon RDS) that you should be able to tie into, see the addons page for a complete list. http://addons.heroku.com/\nSetting up Django Caching on Herku # To use memcache on Heroku, I went to the add-on page and installed the free memcache addon. Once I did this I can run the heroku config command to find out my config settings. I then use that information in my django settings file so that I can have access to caching in my app.\n$ heroku config ... MEMCACHE_PASSWORD =\u0026gt; xxxxxxxxxxxx MEMCACHE_SERVERS =\u0026gt; instance.hostname.net MEMCACHE_USERNAME =\u0026gt; xxxxxxxxxxxx ... They also offer redis if you prefer that.\nSending Email from django on Heroku # If you need to send or receive email in your application there are a few email add-ons that you can use. Pick the one you want and then run the \u0026lsquo;heroku config\u0026rsquo; command described above to get your settings, and update your django settings accordingly.\nApplication size # You application and all of it\u0026rsquo;s dependencies can\u0026rsquo;t be more then 100MB in size.\nHeroku Pricing # Directly from this page: http://devcenter.heroku.com/articles/how-much-does-a-dyno-cost\nDynos cost $0.05 per hour, prorated to the second. For example, an app with four dynos is charged $0.20 per hour for each hour that the four dynos are running.\nPricing is based on calendar time. If you set your app to four dynos, you will be charged $0.20 per hour regardless of the traffic your site serves during that time.\nEach application receives 750 free dyno hours per month. For example if you have 1 web dyno running for all of April, and a worker dyno running half the time you would have 330 dyno-hours billed that month or $16.50 (720 web dyno hours + 360 worker dyno hours - 750 free dyno hours).\nSee this page for more details: http://www.heroku.com/pricing#0-0\nConclusion # I haven\u0026rsquo;t really had much time to really play with Heroku, and stress test it at all, but I have to say it is pretty impressive to begin with. There are still some rough edges, but I\u0026rsquo;m sure they will have those smoothed out before they remove the beta tag. The thing that impresses me the most is all of the addons that you have access to out of the box. I\u0026rsquo;m not sure how many of these work with the new platform, or with django right now, but I\u0026rsquo;m sure it is only a matter of time before they are available.\nI would highly recommend signing up and trying out their service, it is free so what do you have to lose?\nUpdate # 2/16/2012: Full disclosure. On Feb 16th 2012, I accepted a job with dotCloud a competitor to Heroku. I plan on keeping this blog post up to date and impartial. If you think there are any errors, please let me know in the comments.\n","date":"November 30, 2011","externalUrl":null,"permalink":"/blog/2011/11/developers-guide-for-running-django-apps-on-heroku/","section":"Posts","summary":"","title":"Developers guide to Running Django Applications on Heroku","type":"post"},{"content":"","date":"September 26, 2011","externalUrl":null,"permalink":"/tags/django-maine/","section":"Tags","summary":"","title":"Django-Maine","type":"tags"},{"content":"When I was at DjangoCon a few weeks ago, it was great hanging out with all of the fellow Djangonauts talking about Django and all of other great technologies. Now that I\u0026rsquo;m home, I have decided to get off my butt and do something I should have done a while ago.\nWithout further ado, I am pleased to announce the creation of DjangoMaine.com, a new Django user group based in Portland Maine. We plan on meeting monthly to discuss Django and other python related technologies. We are currently looking for members, and topic ideas for future meetings. If you have any ideas or you are interesting in hearing more, check out our website http:///www.DjangoMaine.com\n","date":"September 26, 2011","externalUrl":null,"permalink":"/blog/2011/09/djangomainecom-django-user-group-portland-maine/","section":"Posts","summary":"","title":"DjangoMaine.com - Django User group in Portland Maine","type":"post"},{"content":"I recently had my 3 year anniversary at CashStar, and it got me thinking about how fast the last 3 years have gone. When I first started at CashStar there were only a handful of people, and we had nothing, and it was great. We had no products, no customers, which might sound bad at first, but we also didn\u0026rsquo;t have any customer, or production support issues to slow us down. We could go into work and put in 8 really productive hours, and feel good about yourself when you left that day.\nOver time, we built our products and signed customers, and slowly over time the company grew to the size we are today, which is around 50 people. I\u0026rsquo;m no where as productive as I was when we started, I\u0026rsquo;m lucky if I can sit at my desk for an hour a day. If I need to get anything done I normally do it at night or on the weekend, which means I end up working a ton of hours in order to get the same productivity level I had when we first started.\nThe funny thing is, even with all of these distractions, I\u0026rsquo;m still much more productive then I ever was when I worked for a big company. Working for a big company felt like I was walking, working for a startup makes you feel like you are running in comparison.\nIn my past I have worked for companies that had 10,000, 2,500 and 800 people respectively. Even though they are very different in size, they all felt like a big company. The more I think about it, the more I realize that I don\u0026rsquo;t think I could ever work for a big company again. Working for a big company has a different feel to it, and you normally have to worry about office politics and other things that you don\u0026rsquo;t have to worry about in a small company. It can be very painful at times when you want to get something done at a big company and you can\u0026rsquo;t.\nI have worked at a few big companies and I have worked with people at other big companies for a while, and one thing I noticed a while ago, is that there are some really smart people working at big companies and there are also some dumb people who work at big companies. You don\u0026rsquo;t see many dumb people at small companies, mainly because there is no place to hide. In a small company you need to carry your own weight, and if you don\u0026rsquo;t it is pretty obvious. In a large company it isn\u0026rsquo;t as obvious to find the slacker.\nI always wondered why the big companies hired the dumb people, and i think I came up with a theory. Basically it boils down to this. When you are a small company you don\u0026rsquo;t have that much work, so you can get by with less people. When you have less people to hire, you want to make sure you get the best of the best, the top 1%. So you wait until great people come along, and you hire those.\nSo you only hire great people, and things are going well, and the company grows. Eventually you will get to the point where you now have all the great people in the local area working at your company. You have a choice to make, you can either lower your standards a little or you can start importing people from other areas, which will cost more money. If you are really successful you can afford to import great talent from all over. Most likely you can\u0026rsquo;t do that, so you lower your standards a little, and you keep growing. As your company gets bigger and bigger eventually you have lowered your standards enough where the occasional dummy will sneak in, and once one dummy gets in, it is all over.\nThe other side of the coin is that as you get bigger, the company needs to add structure and process to control all of the less then great people, which makes it less fun for the great people, who start to leave the company. Most companies don\u0026rsquo;t mind at this point because all of the products that the great people have built are already finished, so they just need to be maintained, and they can do that with cheaper (not as smart or experienced) people. The managers in charge of these products look like heros because they saved some money on their budget due to the lower salaries, and they get promoted. It is just a matter of time before the company slows to a crawl and stops being productive.\nNow of course what I\u0026rsquo;m saying is a little over dramatic, and not all big companies are like this, but there is some truth to what I\u0026rsquo;m saying, and if you ever worked at a big company or with a big company, you know what I\u0026rsquo;m talking about. I\u0026rsquo;m not saying that only dummies work in big companies, because if that were true those companies would go out of business very quickly. I know a lot of really smart people who work at big companies, and they are happy.\nNow that I say I would never want to work for a big company again, I guess there would need to be a handful of exceptions to the rule. Here is my list of big companies I might want to work for and my reasons. Of course I reserve the right to change my mind if the right company comes along, with the right opportunity.\nAmazon - They aren\u0026rsquo;t just an e-commerce company, they are actually a technology company as well. They have built their own products (kindle) and have their own services (AWS). They are innovators and trend setters and that sounds like a cool company to work for.\nNetflix - They revolutionized the movie rental market, and are now the leaders in streaming movies. They handle something around 30% of all internet traffic on a given night, which is unreal. I have also heard they also have great benefits.\nApple - Probably the best product company in the last 100 years. They make cool stuff, that everyone loves. I\u0026rsquo;m sure they work you hard to get everything perfect, but to be able to say I helped built that new gizmo that someone was using, would be priceless.\nESPN - I love sports, I love everything about them, working at ESPN would be a dream job.\nRackSpace - Even though they are hosting company, they are a trend setting hosting company and they are building products as well. They are driving force behind OpenStack, and they have some really cool cloud server products as well. The whole company looks like it would be fun to work at.\nI\u0026rsquo;m sure there are a few more companies that I missed, if I remember I will be sure to update this post. Leave a comment with the big company you wouldn\u0026rsquo;t mind working for, and why, and see if you can convince me to add it to my list.\n","date":"August 31, 2011","externalUrl":null,"permalink":"/blog/2011/08/ill-never-work-for-a-big-company-again/","section":"Posts","summary":"","title":"I'll never work for a big company again","type":"post"},{"content":"","date":"August 31, 2011","externalUrl":null,"permalink":"/tags/startup/","section":"Tags","summary":"","title":"Startup","type":"tags"},{"content":"","date":"August 12, 2011","externalUrl":null,"permalink":"/tags/clickstream/","section":"Tags","summary":"","title":"Clickstream","type":"tags"},{"content":"Track a visitors movements across your site.\nPossible uses:\nSee what they were doing before they got an error. Use it for analytics and usability testing to see how people are moving around your site, and where you are losing people. Inspired by OpenSymphony\u0026rsquo;s Java project of the same name. http://code.google.com/p/clickstream/\nRequirements # Django 1.1 or newer Django sessions need to be enabled Django south 0.7 or newer for migrations Settings # Custom Settings # Add the following properties to your settings file if you want to override the default settings.\nCLICKSTREAM_IGNORE_PATHS # Default = ()\nTakes a list of regular expressions for paths that you want to ignore if it is picked up by these clicks won\u0026rsquo;t be tracked.\nCLICKSTREAM_IGNORE_PATHS = (\u0026rsquo;^/private/\u0026rsquo;, \u0026lsquo;^/dont-track-me/\u0026rsquo;, \u0026lsquo;super-secret.txt\u0026rsquo;)\nCLICKSTREAM_IGNORE_ANONYMOUS # Default = False\nIf you only want to track people that are logged in, change to True\nCLICKSTREAM_IGNORE_ANONYMOUS = False\nCLICKSTREAM_IGNORE_IP # Default = ()\nA list of IP\u0026rsquo;s that you don\u0026rsquo;t want to track\nCLICKSTREAM_IGNORE_IP = (\u0026lsquo;127.0.0.1\u0026rsquo;, \u0026lsquo;10.10.10.10, \u0026lsquo;8.8.8.8\u0026rsquo;)\nMiddleware # Add \u0026lsquo;clickstream.middleware.ClickStreamMiddleware\u0026rsquo; to your MIDDLEWARE_CLASSES settings it needs to be after \u0026lsquo;django.contrib.sessions.middleware.SessionMiddleware\u0026rsquo; in the list.\nApp # Add clickstream to INSTALLED_APPS\nSync Tables + Migrate # Run \u0026lsquo;./manage.py syncdb\u0026rsquo; to load up the tables in your database.\nRun \u0026lsquo;./manage.py migrate\u0026rsquo; to run the south migrations\n","date":"August 12, 2011","externalUrl":null,"permalink":"/blog/2011/08/django-clickstream/","section":"Posts","summary":"","title":"Introducing django-clickstream","type":"post"},{"content":"","date":"June 27, 2011","externalUrl":null,"permalink":"/tags/centos5/","section":"Tags","summary":"","title":"Centos5","type":"tags"},{"content":"","date":"June 27, 2011","externalUrl":null,"permalink":"/tags/fabric/","section":"Tags","summary":"","title":"Fabric","type":"tags"},{"content":"","date":"June 27, 2011","externalUrl":null,"permalink":"/tags/nginx/","section":"Tags","summary":"","title":"Nginx","type":"tags"},{"content":"When I first started working with django I deployed my apps using apache and mod_python. Then after a little while I started playing with nginx and switched my setup so that nginx was serving the static content and reverse proxied requests back to apache and mod_python. Not too long after that, I switched out mod_python with mod_wsgi and ran mod_wsgi in daemon mode.\nThis setup worked well for a while, but one thing I never really liked was the fact that I needed to run apache which is pretty heavy even when you strip out all the unused modules. Apache is great, but all I was really using it for was a router between nginx and mod_wsgi, I wasn\u0026rsquo;t using any of the other features in apache.\nI looked at fastcgi and uswgi and they looked good, but for one reason or another I never made the jump. Recently I have been hearing a lot of good things about green unicorn, so I decided to check it out. When I first looked at it, it was fairly new and because of this a little concerned with stability, so I kept and eye on it and watched it mature.\nWhile I was waiting for green unicorn to mature I ended up doing a lot of research on the new python hosting services that recently hit the market. Three out of the five services that I looked were using green unicorn, the other two were using uWSGI.\nThe fact that these three services are basing there new businesses on green unicorn gave it a lot of credibility. Not too long after that I started playing with green unicorn to see what it would take to get my sites up and running.\nThe first thing that I noticed was that I didn\u0026rsquo;t need to create a wsgi file if I used their gunicorn_django command, which was pretty sweet. The fact that they built it into the service shows you that django is a first class citizen.\nThe second thing that I noticed was that I needed a way to start up green unicorn and keep it running, something that apache does for you with mod_wsgi. I did a little bit of research and found out that supervisord would work perfectly for what I needed to do with green unicorn.\nBecause seeing is better then reading, I\u0026rsquo;ll guide you throw the steps that you will need to do in order to get your system setup in a way that will make using green unicorn very easy, especially if you want to run more then one website on your server. I\u0026rsquo;m going to use a 256MB rackspace cloud instance running centos 5.5.\nCreate a rackspace cloud server # Go into the rackspace cloud server management website and allocate yourself a new 256MB CentOS 5.5 server or if you prefer do the same thing using their API. Now that you have a server and the root password, follow along step by step to get you system all setup.\nSoftware and versions # RackSpace Cloud Server 256MB CentOS 5.5 Python 2.6.6 nginx 1.0.4 supervisord 3.0a10 virtualenv 1.6.1 pip 1.0.1 gunicorn 0.12.2 fabric 1.1.0 Bitbucket project # To make things easier I have created a django bootstrap project directory with all of the file used in the blog post. It is located here, so feel free to clone and fork.\nhttps://bitbucket.org/kencochrane/django-gunicorn-nginx-supervisord-bootstrap/\nLogin to server # ssh root@\u0026lt;RackSpaceIP\u0026gt; Update packages # yum -y update Install packages # You might not need all of these right now, but I normally need these down the line, so doing them all now.\nyum -y install emacs readline-devel ncurses-devel libevent-devel glib2-devel libjpeg-devel freetype-devel bzip2 bzip2-devel bzip2-libs openssl-devel pcre pcre-devel gpg make gcc yum-utils unzip Add a django user as a system user # useradd -d /opt/django -m -r django Set password for django to what ever you want # passwd django Setup directories # mkdir -p /opt/django mkdir -p /opt/django/apps mkdir -p /opt/django/logs mkdir -p /opt/django/logs/nginx mkdir -p /opt/django/logs/apps mkdir -p /opt/django/configs mkdir -p /opt/django/scripts mkdir -p /opt/django/htdocs mkdir -p /opt/django/tmp mkdir -p /opt/django/configs/nginx mkdir -p /opt/django/configs/supervisord mkdir -p /opt/django/apps/my_app Add blank html page # echo \u0026#34;\u0026lt;html\u0026gt;\u0026lt;body\u0026gt;nothing here\u0026lt;/body\u0026gt;\u0026lt;/html\u0026gt; \u0026#34; \u0026gt; /opt/django/htdocs/index.html Install Zlib # # download from zlib.net mkdir -p /tmp/downloads cd /tmp/downloads wget http://www.zlib.net/zlib-1.2.5.tar.gz tar -xvzf zlib-1.2.5.tar.gz cd zlib-1.2.5 ./configure -s make install Install python 2.6.6 # CentOS 5.5 doesn\u0026rsquo;t come with python2.6 pre installed so we need to do that on our own.\nmkdir -p /tmp/downloads cd /tmp/downloads wget http://www.python.org/ftp/python/2.6.6/Python-2.6.6.tgz tar -xvzf Python-2.6.6.tgz cd Python-2.6.6 ./configure --enable-shared make make altinstall Add the following to /etc/profile # We need to add the lib path to the LD_LIBRARY_PATH or else you will get an error saying it can\u0026rsquo;t find libpython2.6.so.1.0\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/:/usr/local/lib64/ Source the new profile # source /etc/profile Install distribute # mkdir -p /tmp/downloads cd /tmp/downloads curl -O http://python-distribute.org/distribute_setup.py python2.6 distribute_setup.py Install Pip \u0026amp; virtualenv # mkdir -p /tmp/downloads cd /tmp/downloads curl -O -k https://raw.github.com/pypa/pip/master/contrib/get-pip.py python2.6 get-pip.py pip install virtualenv Install supervisor # pip install supervisor Install mercurial # pip install mercurial Install NGINX # mkdir -p /tmp/downloads cd /tmp/downloads wget http://nginx.org/download/nginx-1.0.4.tar.gz tar -xzvf nginx-1.0.4.tar.gz cd nginx-1.0.4 ./configure --sbin-path=/usr/local/sbin --with-http_ssl_module --with-http_stub_status_module make /etc/init.d/nginx stop sleep 2 sudo make install sudo chmod +x /usr/local/sbin/nginx Install my app # Add first virtualenv # cd /opt/django/apps/my_app/ virtualenv --distribute --no-site-packages v0.1 # make this a post_create hook? touch /opt/django/apps/my_app/v0.1/.venv cd /opt/django/apps/my_app/v0.1/ hg clone https://bitbucket.org/kencochrane/django-gunicorn-nginx-supervisord-bootstrap my_app ln -s /opt/django/apps/my_app/v0.1 /opt/django/apps/my_app/current ln -s /opt/django/apps/my_app/current/my_app/conf/nginx.conf /opt/django/configs/nginx/myapp.conf ln -s /opt/django/apps/my_app/current/my_app/conf/supervisord.conf /opt/django/configs/supervisord/myapp.conf # activate the ve source /opt/django/apps/my_app/current/bin/activate cd /opt/django/apps/my_app/current/my_app/ ./bootstrap.py Configure nginx # # as root mkdir -p /etc/nginx ln -s /opt/django/apps/my_app/current/my_app/server/etc/nginx.conf /etc/nginx/nginx.conf ln -s /usr/local/nginx/conf/mime.types /etc/nginx/mime.types ln -s /opt/django/apps/my_app/current/my_app/server/init.d/nginx /etc/init.d/nginx chmod 755 /etc/init.d/nginx chkconfig --add nginx chkconfig nginx on Configure Supervisord # # as root ln -s /opt/django/apps/my_app/current/my_app/server/etc/supervisord.conf /etc/supervisord.conf ln -s /opt/django/apps/my_app/current/my_app/server/init.d/supervisord /etc/init.d/supervisord chmod 755 /etc/init.d/supervisord chkconfig --add supervisord chkconfig supervisord on Firewall # We need to open up the firewall so that we are allowed connection, if you don\u0026rsquo;t know anything about this, check out these links.\nhttp://cloudservers.rackspacecloud.com/index.php/Firewalls http://cloudservers.rackspacecloud.com/index.php/Introduction_to_iptables http://cloudservers.rackspacecloud.com/index.php/Sample_iptables_ruleset\n# Open http port 80 iptables -I RH-Firewall-1-INPUT -p tcp --dport 80 -j ACCEPT .bashrc file changes # I can\u0026rsquo;t remember where I saw this little trick, if you know please let me know so that I can give them credit. If you put a file in your mercurial directory called .venv, when you cd into the directory this little bash hack will automatically activate your virtual environment for you. This allows you to have something similar to virtualenvwrapper in this custom setup.\nAdd this code to the .bashrc file\nemacs /opt/django/.bashrc # # User specific aliases and functions has_virtualenv() { if [ -e .venv ]; then deactivate \u0026gt;/dev/null 2\u0026gt;\u0026amp;1 source bin/activate fi } venv_cd () { cd \u0026#34;$@\u0026#34; \u0026amp;\u0026amp; has_virtualenv } alias cd=\u0026#34;venv_cd\u0026#34; #end of changes # source the file to get new changes in active shell source /opt/django/.bashrc Change permissions of the django home directory to django # This cleans up and left over root ownership\nchown -R django:django /opt/django/* Switch to django user # su - django Start up nginx # service nginx start Startup supervisord # service supervisord start Test Nginx and supervisord # Check supervisord status\nsupervisorctl status my_app RUNNING pid 13594, uptime 0:00:05 To check nginx go to the IP or domain name for your rackspace server in your browser and make sure it worked.\nUpdating the application using fabric # Inside of the bitbucket project directory there is a file called fabfile.py. This file will allow you to update the application from your machine whenever you want just by calling one command.\nIt will prompt you for your hostname and password for the django user. Then it will go out to the rackspace server and pull and update the app and restart the application in supervisord. It is very basic for right now, but should get you started if you want to do more advanced stuff.\nfab update_server Conclusion # Now that we have everything setup, if you want to add a new application to our setup all we need to do is.\ncreate a new directory under apps create the virtualenv run the bootstrap to install the software make sure that the application has a supervisord and nginx configuration file symlink those files to the correct locations in the config directory run any python management commands you might need to run (syncdb, migrate, etc) reload supervisord and nginx you should be good to go. I hope this was helpful to someone besides myself, if it was helpful for you please let me know in the comments.\n","date":"June 27, 2011","externalUrl":null,"permalink":"/blog/2011/06/django-gunicorn-nginx-supervisord-fabric-centos55/","section":"Posts","summary":"","title":"Setting up Django with Green Unicorn, nginx, supervisord and fabric on CentOS 5.5","type":"post"},{"content":"","date":"June 27, 2011","externalUrl":null,"permalink":"/tags/supervisord/","section":"Tags","summary":"","title":"Supervisord","type":"tags"},{"content":"","date":"June 6, 2011","externalUrl":null,"permalink":"/tags/apphosted/","section":"Tags","summary":"","title":"Apphosted","type":"tags"},{"content":" Updates # Updated 2/16/2012: Full disclosure. On Feb 16th 2012, I accepted a job with dotCloud. I plan on keeping this blog post up to date and impartial. If you think there are any errors, please let me know in the comments below. Updated 1/26/2012: Updated Gondor.io with info from Donald Stufft. Updated 01/24/2012: changed gondor.io to include SSL support Updated 01/01/2012: Added Heroku and updated ep.io, gondor.io, and dotcloud. Intro # For the past 6 weeks I have been trying out all of the new django/python hosting services that are currently available today, and I have been writing about my experiences along the way. It only makes sense to conclude this series of articles with one last article comparing all of the services against each other. It is important to note that many of these services are still in development and aren\u0026rsquo;t even available to the general public, so I\u0026rsquo;ll try to keep this article up to date as these services change over time. With that said, don\u0026rsquo;t take my word for it, go out and try all of these services on your own and find out which one you like the best, you won\u0026rsquo;t be disappointed.\nQuick Recap # ep.io # Blog post: My Experiences with ep.io\nPretty solid offering, with a nice set of features and a decent price. Good set of documentation.\n[Update: Epio closed down on May 31st 2012 ]\ngondor.io # Blog post: My day in gondor.io\nTheir website has a nice list of features that they expect to have once they officially launch, but most of those features aren\u0026rsquo;t available yet. The documentation is a little light, but the service has a lot of potential.\ndotCloud.com # Blog post: Deploying my Django application to dotcloud.com\nThey have a ton of money ($10M), and with it, a ton of features. They are very developer friendly, but you need to be pretty technical, it isn\u0026rsquo;t quite ready for beginners. Good set of documentation.\n[Full disclosure: I now work for dotCloud]\nAppHosted.com # Blog post: apphosted.com Django Hosting Review\nBuilt with security, stability and scalability on their mind from the beginning. They have a solid offering, and they look to be targeting enterprise customers who will pay more for that peace of mind. They are currently lacking some key features, which I\u0026rsquo;m sure they will be adding soon. Good set of documentation.\nDjangoZoom.com # Blog post: DjangoZoom.com Review\nBuilt to be fast and easy to use, still missing some key features, but I\u0026rsquo;m sure they will be available by the time they go live. Decent amount of documentation.\nHeroku.com # Blog post: Developers guide for running django apps on Heroku\nOriginally built as a Ruby on Rails service, they have now added support for python. The service is still in beta, but it has a lot of potential. It looks a lot like what dotCloud is offering, with the ability to run code from almost any language, and a ton of different add-ons for lots of different services.\nAwards # Fastest Deployments: # DjangoZoom.com - Zoom is right, it doesn\u0026rsquo;t take long to configure your application and have it up and running on their servers.\nEasiest To Use: # DjangoZoom.com - No command line interface needed, just a web browser, a project in a git repo that it has access too, and a few questions answered and your app is up and running.\nMost Features: # dotCloud.com - They raised $10 Million, and they are spending it on hiring people, buying companies, and building services. They have a bunch of services currently and they don\u0026rsquo;t plan on stopping, their roadmap has everything on it, and I think if you give them enough time they will eventually have a service for everything.\nMost Developer Friendly: # dotCloud.com - They give lots of features without handcuffing the developer. They are the only company with full database and shell access. They do a good job of offering the same type of service a developer could get if they built it themselves.\nBest Overall Value: # ep.io - This one is hard since most of the services don\u0026rsquo;t have any pricing listed, but currently ep.io is in the lead, they offer reasonable prices with a nice Free tier. This allows developers to try out the service for FREE, as well as run smaller pet projects that they might not have tried before because they didn\u0026rsquo;t want to pay for hosting. They have the second most features available, second to only dotCloud, and there service is pretty solid.\nEasiest Project Setup: # apphosted.com - Their goal was to make it real easy to get your project up onto their servers without having to change your project, and they did that, there was very little if any changes I needed to make in order to get my application up on their servers.\nDjango Feature Hosting Matrix # To make things easier when comparing all of the different services I have built this matrix with all of the information I compiled from each of the services. I don\u0026rsquo;t know the answers for all services, but I\u0026rsquo;ll update it when I find out those answers. If you know the answers feel free to post a comment to let me know.\nI have included all of the new django services as well as google app engine and webfaction.com, a tradition hosting service. This should make it a little easier to see how these new services compare to other hosting options.\n[Chart removed since it was out of date]\nWho wins? # It is really hard to pick just one winner, mainly because most of these services are still in beta and not quite finished yet. Also because each service is a little different, and it will depend on what you are trying to do. So, go out try them out, and let me know which one you picked.\nThe real winner is us, the developers, and the python community in general. We now have a bunch of really cool services that will make our lives better, and that is awesome.\nThank you! # I want to thank all of the people who made these services, and gave me early access to their systems so that I could play around with them. I wish them the best of luck, and hope they all are really successful, and they are around for a long time.\n","date":"June 6, 2011","externalUrl":null,"permalink":"/blog/2011/06/django-hosting-roundup-who-wins/","section":"Posts","summary":"","title":"Django Hosting Roundup: (Ep.io vs Gondor.io vs DotCloud vs Heroku vs AppHosted vs DjangoZoom) Who wins?","type":"post"},{"content":"","date":"June 6, 2011","externalUrl":null,"permalink":"/tags/djangozoom/","section":"Tags","summary":"","title":"Djangozoom","type":"tags"},{"content":"","date":"June 6, 2011","externalUrl":null,"permalink":"/tags/epio/","section":"Tags","summary":"","title":"Epio","type":"tags"},{"content":"","date":"June 6, 2011","externalUrl":null,"permalink":"/tags/gondor/","section":"Tags","summary":"","title":"Gondor","type":"tags"},{"content":"This is part five in my series on django hosting services. Previously, I looked at ep.io, apphosted.com, gondor.io, dotcloud.com and now I\u0026rsquo;m looking at DjangoZoom.com.\nDjangoZoom.com is the brain child of Nate Aune and Shimon Rura and is based in Boston, Massachusetts. It was founded in 2010 at StartupWeekend Boston and was a finalist in the MassChallenge. Their office is in the Dogpatch Labs space for startups in Cambridge, Massachusetts. They are currently still in a closed beta, but they were nice enough to send me an invite to check it out.\nNormally when I check out a service for the first time, I look over the documentation to see what it can do, and what it can\u0026rsquo;t do, and what I need to do in order to get my app up and running. DjangoZoom has a nice collection of documents that help guide you through the process of getting your application up and running on their platform. You need to be logged in, in order to read the documents, so I won\u0026rsquo;t be able to link to any documents here, but if you are lucky enough to get an invite to DjangoZoom, I would check out the documents first, they will make the process much easier. They answer most of the common questions that you will have, and if you can\u0026rsquo;t find your answer you can send them a message and they are very fast to respond.\nAfter reading the documents I learned a few things. First off, DjangoZoom is a little different then the other services, all of the previous services that I have reviewed ep.io, dotcloud, apphosted, gondor.io, had a command line client that you use to push your application up onto their servers.\nDjangoZoom doesn\u0026rsquo;t have a command line client at all, they only have a web based control panel that you use to install and configure your application. Also, instead of pushing your application code up on to their servers, they pull the code down from your code repository at build time. Most of the other services try and make it so that you can run any python/wsgi based application, where as DjangoZoom, is only supporting Django applications.\nNot having a command line client has it\u0026rsquo;s pros and cons. The main advantage of not having a command line client is simplicity. No need to install anything on your machine in order to deploy your code, all you need is a web browser. You can deploy your code from anywhere you have an internet connection. Making the application deployment as easy as possible will open up their service to people that are less technical, and might not be comfortable installing a command line client and running commands. The disadvantages of using just a web based console is that you are limited to what you can do in a browser. This is going to be less and less of an issue going forward as web browsers are getting more and more powerful all of the time. Also, just because they only offer a web based console now, doesn\u0026rsquo;t mean they can\u0026rsquo;t add a command line client later.\nJust like all of the other reviews I\u0026rsquo;m going to test out the system by deploying my blog onto their system and see how it goes. My blog is currently hosted on bitbucket.org which uses mercurial, and currently djangozoom.com only offers support for git. In order to use djangozoom.com I needed to make a fork of my blog and put it on github. You can find my fork here: https://github.com/kencochrane/kencochrane_blog . They recommend github for git hosting, but you can use just about any git repo that is publicly accessible. If you have a private git repo, and you are on github just add the user djangozoom as a collaborator on your project. If you aren\u0026rsquo;t on github and you have a private repo, you can add their public key to your repo, and it will give them permission to access your git repository from their servers.\nInstall Process # Once I had my code up on github I needed to make a few changes in order for it to work correctly, nothing major just needed to make my settings.py file a little more standard, which was pretty typical for the other services as well. Now that I had that done, I was ready to login to DjangoZoom and setup my application. Registering and logging into DjangoZoom was pretty straight forward, it was the same process as most online services, so just about anyone could do it with their eyes closed.\nAfter I logged in, I was able to add my application, here are the steps that I took to get my application up and running.\nClicked on \u0026ldquo;add a new project\u0026rdquo;\nPut in repo url: git://github.com/kencochrane/kencochrane_blog.git\n(Only SSH authentication is supported, so use either an unauthenticated public URL, or an SSH url (e.g. git@github.com:yourname/project.git. On github.org, use the Git Read-Only URL if your repo is public, or the SSH URL if private.)\nHit Next.\nEntered my title \u0026ldquo;kencochrane_blog\u0026rdquo;\nselected django 1.3\nsettings module : mysite.settings\nstatic media location : \u0026ldquo;static /site-media\u0026rdquo;\nno additional directories to add to python path\nput in my requirements.txt file, so that it knows where to find my pip requirements. You don\u0026rsquo;t need to do this, djangozoom will scan your projects and find this out for you, but it didn\u0026rsquo;t work for me, I have an issue with my application that I wasn\u0026rsquo;t able to find, and it was causing issues with their autoscanner. I think this is more my fault then theirs.\nnot putting in any extra requirements.\nbase(repo) python package: put mysite.settings because my settings.py and root urls.py are at the top level of your repository. normally I don\u0026rsquo;t do this, but I was following the example of this repo https://github.com/natea/mezzanine-site\nChecked the \u0026ldquo;ready to roll, deploy my project now\u0026rdquo; check box and hit next\nAt first I had some errors with my requirements, I needed to remove django and mysql and update the python-twitter lib. Once I did that, it worked like a charm. The whole deploy process only took 2 minutes and 45 seconds.\nWow that was fast, pretty impressive considering it needed to pull down all of the code from github, and download a ton of dependencies, and then do what ever it needs to do to set it up on to their servers. After talking with them about this, they said they hope to get it even faster, which would be pretty impressive.\nNow that I have my app deployed I can change some of the settings.\nI created an alias to for my app to https://kencochrane.djangozoom.net\nYou can also add a hostname like kencochrane.net.\nyou can add a wildcard hostname as well. *.kencochrane.net.\nYou can create a django superuser.\nYou can also run any of your manage.py commands right from the web console.\nThis whole process was pretty straight forward and didn\u0026rsquo;t take long to complete. Now that I have done it with one application, it will be much easier for the next one.\nCool Feature Request # Currently DjangoZoom makes it really easy to deploy your application to their servers. What would be cool is if they would make it easy to deploy common django applications to their service. For example on their blog they have a video to show how easy it is to deploy django-cms to DjangoZoom.\nIn the video it shows them going out to github, forking and copying the link to the project, and then making some minor changes on the djangozoom settings page to get it to work correctly, and then it deploys. These steps weren\u0026rsquo;t complicated but you needed to know what to do in order to get it to work. What if they made this a one step process?\nClick on a \u0026ldquo;install django-cms\u0026rdquo; link, and then their service would do what it needs to do to get it up and running on their service, and then returns you to a page where you can configure the app, add a django admin user, change the url, etc. That would allow someone who doesn\u0026rsquo;t know anything about django an easy way to get a django-cms based website in no time. They could add one click installs for all kinds of common django projects, and expand outside of the developer community, which would expand their customer base. Those customers are the ones that really don\u0026rsquo;t want to worry about server deploys and server management, and they are the ones that will pay more for those features.\nI\u0026rsquo;m sure this feature isn\u0026rsquo;t on their roadmap, but I just thought I would throw it out there and see what people have to say about it.\nDjangoZoom.com status # DjangoZoom is still in closed beta, which means they are not done with it yet, and there may be some occasional bugs that pop up here and there, but that is expected. Another thing about a beta is that not all of the features are finished yet. So before you jump in and try and get your application up and running, check and make sure that your application can run correctly on the service with the features they have available so far. Here is a run down of all the different features your application might need and where they stand in DjangoZoom today.\nLogging # You currently can\u0026rsquo;t view your django or nginx logs at this time, they are working on this feature and hope to have it so that you can view your logs in the web based console in the future. In the meantime you can use a tool like django-sentry to view your django-logs.\nSSL # Not available yet.\nCaching # They currently don\u0026rsquo;t support a caching server, they want to make sure they set it up correctly so that it is reliable and so that it is secure. In the meantime you can use django local memory caching.\nDatabase # They support PostgreSQL 8.4, if you are using mysql you will need to convert your app to support PostgreSQL. Since the Django ORM handles this by default, this shouldn\u0026rsquo;t be a big deal unless you did something custom. No direct database access available. Loading data into system only supported by using django dumpdata/loaddata\nBackground tasks # Not available yet, celery support possible in the future.\nCronjobs # Not supported, they recommend Advanced Python Scheduler or django-cron which should run on their system.\nShell Access # Not supported\nAPI # Not available yet, one planned for the future.\nCommand line client # Not available yet, one planned for the future. Use the web base control panel.\nVersion control # Currently only supports git.\nVideos # Here are some videos of the DjangoZoom.com deployment process.\nConclusion ---------- I really enjoyed playing with DjangoZoom, it is quick and easy to use, it is still missing some key features that some people will need, but I\u0026rsquo;m sure they will add those soon enough. I wish them luck, and I can\u0026rsquo;t wait to see them go live to the general public.\nUpdate: # Read how this service stacks up against other services like it in my Django hosting roundup\n2/16/2012: Full disclosure. On Feb 16th 2012, I accepted a job with dotCloud a competitor to DjangoZoom. I plan on keeping this blog post up to date and impartial. If you think there are any errors, please let me know in the comments.\n","date":"June 5, 2011","externalUrl":null,"permalink":"/blog/2011/06/djangozoom-com-review/","section":"Posts","summary":"","title":"DjangoZoom.com Review","type":"post"},{"content":"This post is the fourth in my series about the new Django hosting services that just recently hit the market. Previously I have reviewed ep.io , gondor.io , and dotCloud.com .This post reviews AppHosted.com , another similar service that is currently in beta. I was lucky enough to get a sneak peak of this service before it was released to the general public, and here are my notes and impressions for the short time I had to play with the new service.\nOverview # AppHosted.com uses a similar concept as the other django hosting services, they provide you with a command line client called metro that you use to interact with the service. Like the other command line clients, it is pretty much just a nice wrapper around their API, so anything you can do with the metro client you can do with their API. The client is used to configure your service and push your files up into their servers. Once the application is created and your code has been uploaded to their server, you can use their web based control panel to configure or manage your application.\nSo that we can compare apples to apples I will go through the process of installing my blog onto the service so that we can see how it works with apphosted compared to the other services. It will also hopefully provide a very simple how to guide on how to install your application onto AppHosted.com.\nFirst Steps # Like all new projects the first thing that I like to do is create a new project directory and a new virtual environment for the project.\n# move into my projects directory $ cd ~projects/ # make my virtual environment apphosted $ mkvirtualenv apphosted # clone my blog $ hg clone https://bitbucket.org/kencochrane/kencochranenet kencochrane_apphosted # move into the new project directory $ cd kencochrane_apphosted Now that I have a local working copy of my blog code and a new virtual environment I\u0026rsquo;m ready to go.\nCommand Line Interface # AppHosted has a command line client called Metro, it uses this client to configure and upload your application onto the apphosted servers. Metro only has three dependencies: httplib2, paramiko, and pycrypto, if you don\u0026rsquo;t have them installed pip should handle that for you.\n# install the metro client $ pip install metroclient Downloading/unpacking metroclient Running setup.py egg_info for package metroclient Requirement already satisfied (use --upgrade to upgrade): httplib2 in /Users/ken/.virtualenvs/apphosted/lib/python2.6/site-packages (from metroclient) Requirement already satisfied (use --upgrade to upgrade): paramiko in /Library/Python/2.6/site-packages (from metroclient) Requirement already satisfied (use --upgrade to upgrade): pycrypto\u0026gt;=1.9 in /Library/Python/2.6/site-packages (from paramiko-\u0026gt;metroclient) Installing collected packages: metroclient Running setup.py install for metroclient changing mode of build/scripts-2.6/metro from 644 to 755 changing mode of /Users/ken/.virtualenvs/apphosted/bin/metro to 755 Successfully installed metroclient Initial Data \u0026amp; Django admin user # At the moment Apphosted.com doesn\u0026rsquo;t provide an easy way to create a django admin user, in order to create this user in your appHosted.com database you need to create it on a local database and then export that data into a fixture called initial_data.json.\n# locally I run this command to dump the data, so that I can load on app hosted. $ python manage.py dumpdata \u0026gt; initial_data.json There is another way to create the django admin user, which I\u0026rsquo;ll describe a little later. This would also be the same way that you would import any application data that you need in order for your application to run correctly.\nMetro Client # One of the goals for appHosted.com was that they didn\u0026rsquo;t want to require that you make lots of changes to your project directory structure in order to use their service. In order to do this they need to do a lot more work to figure out what it is you want to do, and if they can\u0026rsquo;t figure it out, they will ask you some questions.\nNow that we have our project directory and our initial_data.json file all ready, lets go ahead and deploy the application. Activate the virtual environment and go into your project directory. Kick off the metro client and answer the questions that pop up.\n# run the metro client. $ metro Metro Application Manager 1.21 Copyright (c) Lumentica, http://www.lumentica.com Application directory (i.e. path to project): ./ Application name: kencochraneblog Application version: 0.1 Application description: Ken\u0026#39;s AppHosted Blog Application modules (extra; comma-separated): South==0.7.3, django-activitysync==0.2.2, django-debug-toolbar==0.8.4, django-disqus==0.3.4, django-memcache-status==1.0.1, django-tagging==0.3.1, django-xmlrpc==0.1.2, feedparser==4.1, httplib2==0.6.0, oauth2==1.2.0, python-memcached==1.47, python-twitter==0.8.1, simplejson==2.1.2, wsgiref==0.1.2, yolk==0.4.1, docutils==0.6, Pygments==1.3.1 WSGI module (optional): Paste config (optional): Fixtures (optional; comma-separated): Sync database and load fixtures? (y/n): y Use SSL? (y/n): n Force SSL: (y/n): n Build complete. Upload to AppHosted? (y/n): y AppHosted Username: kencochrane AppHosted Password: Deploying... :: Configuring application environment... :: Configuring application permissions... :: Configuring application server... :: Application deployed to server successfully... :: Application deployment complete... What did this do for us? It takes our application, and uploads it to their server, and then it installs all of our dependencies that we listed above. Then it configures it according to our answers, and tries to start it up, if we told it to, it will sync our database and load our fixtures. It also takes all of this information and stores it in a file called metro.config, so that we don\u0026rsquo;t need to type that in every time if nothing has changed.\nYou might have noticed that I needed to manually enter in all of my dependencies when prompted, when I asked about this, I was told that if I didn\u0026rsquo;t enter all of the dependencies it would scan the project for a pip style requirements.txt file, and use that to load all of your dependencies. I haven\u0026rsquo;t had a chance to test that out yet, since I already entered my requirements by hand, and once you do it, you don\u0026rsquo;t need to do it again.\nMy blog uses south for migration management, and at the time of this writing there is a bug, and it isn\u0026rsquo;t correctly kicking off the migrate command during the database sync process. They are aware of the issue and they are working on a fix, but there is a work around in the meantime. I\u0026rsquo;ll describe the work around shortly.\nOnce your application is deployed it will almost immediately be available at http://\u0026lt;your-app-name\u0026gt;.apphosted.com.\nLog Files # In order to view the log files, you need to log into the web control panel and from there you can download the logs that you are interested in. At the moment they only support downloading the log files, which is a little inconvenient, it would be nice to have the ability to tail the logs, or see the last 50 or 100 lines in the control panel. I asked about this, and they agreed and said it is on their list of things to work on.\nShell Access # Metro gives you the ability to shell into your application directory, so that you can kick off django management commands, and other cool things. One important thing to note is that when you login to the shell, it is a jailed shell, and you only have access to your own project, and with limited command access. This is done for security reasons. It is nice to know that your application runs in it\u0026rsquo;s own jailed area, and no one else\u0026rsquo;s applications can access your information.\nSetting up the shell access requires a few steps. Follow these directions to get the full details: http://docs.apphosted.com/apphosted/apphosted_shell.html , but basically it requires the following.\ncreate a ssh key if you don\u0026rsquo;t already have one. Login to the web console and add your public key. In the web console browse over to your application Settings-\u0026gt;Tools page and click on the “Update Public Keys” button. (Don\u0026rsquo;t forget about this step, or you won\u0026rsquo;t be able to login. Use the metro client to open the shell. Note: It would be real cool if we didn\u0026rsquo;t have to do step 3, why can\u0026rsquo;t it automatically update the public keys for all apps? I\u0026rsquo;m guessing it is a security reason. The first time, I didn\u0026rsquo;t hit the update public keys button and I couldn\u0026rsquo;t figure out why it wasn\u0026rsquo;t working, and luckily one quick email to the support line, and they let me know what I did wrong, and I was up and running in no time.\nThis is how you can kick off the shell, there are a bunch of command line args that you can pass as well, just do metro \u0026ndash;help and it will show them all to you.\n$ metro -s Note: There is currently a bug that is preventing me from logging into the shell from my MacBook Pro OS X 10.6.7 using Python 2.6.1 . When I do it gives me this error. I have informed appHosted, and they said it looks like an issue with paramiko and python 2.6.1 on the Mac, not sure if it affects other python versions on other operating systems. They are working on the issue.\n$ metro -s Metro Application Manager 1.21 Copyright (c) Lumentica, http://www.lumentica.com Application name: kencochraneblog AppHosted Username: kencochrane AppHosted Password: Traceback (most recent call last): File \u0026#34;/Users/ken/.virtualenvs/apphosted/bin/metro\u0026#34;, line 75, in run_shell ssh.connect(host, username=app_name, port=port) File \u0026#34;/Library/Python/2.6/site-packages/paramiko/client.py\u0026#34;, line 278, in connect for (family, socktype, proto, canonname, sockaddr) in socket.getaddrinfo(hostname, port, socket.AF_UNSPEC, socket.SOCK_STREAM): gaierror: [Errno 8] nodename nor servname provided, or not known Error: [Errno 8] nodename nor servname provided, or not known Work Arounds # Now that you know how to shell into your application, you can also kick off django management commands. I mentioned earlier that there was another way to create the django admin user, well here is how you do it. Login to the metro shell and then kick off the createsuperuser django management command. Logging into the shell and finding things isn\u0026rsquo;t easy do to the added security so to save you some trouble this is what I did. Python isn\u0026rsquo;t in your path so you need to put the fully qualified path to the virtual env\u0026rsquo;s python. Not sure if the jailed shell allows you to edit your .profile if so, it might be nice to configure this so that you don\u0026rsquo;t always have to do the full path.\n# Create superuser (django admin) $ /ve/kencochraneblog/bin/python /app/kencochraneblog/mysite/manage.py createsuperuser # kicking off south migrations by hand. $ /ve/kencochraneblog/bin/python /app/kencochraneblog/mysite/manage.py migrate My application\u0026rsquo;s name is kencochraneblog, so make sure you replace my application name with yours, it should be something like this. replace app_name,django_project_dir and command with your variables.\n$ /ve/\u0026lt;app_name\u0026gt;/bin/python /app/\u0026lt;app_name\u0026gt;/\u0026lt;django_project_dir\u0026gt;/manage.py \u0026lt;command\u0026gt; Background Tasks # AppHosted also supports background tasks via celery, they don\u0026rsquo;t have the documentation created yet, but it should be available soon. They are also planning on supporting a cron like service, not sure if it will be cron, or something similar, but it should do, what you need it to do.\nAPI # AppHosted provides a pretty nice API that allows you to do anything that the metro client does. This will allow you to write your own automation and management scripts to interact with your application. If you practice continuous deployment, it will allow you to automatically deploy the latest version of code on a post commit hook, if all of your tests pass.\nServices # Right now they only support memcache for caching, and PostgreSQL for databases, but they plan on adding more services as they move forward.\nConclusion # I have been playing with AppHosted for a few weeks now, and I\u0026rsquo;m pretty happy with what they have so far. They still have a few issues to work out and they are still a work in progress, but I have been seeing improvements all of the time, so it will only be a matter of time before they are ready for the general public. Every time I came across an issue, they were quick to reply to my emails and solve my issues, or tell me where I was going wrong. If you have any more questions I recommend checking out their documentation @ http://docs.apphosted.com/index.html . Check them out, and make sure you let me know what you think.\nUPDATE:\nThey are now open to the general public.\nUpdate: # Read how this service stacks up against other services like it in my Django hosting roundup\n2/16/2012: Full disclosure. On Feb 16th 2012, I accepted a job with dotCloud a competitor to apphosted. I plan on keeping this blog post up to date and impartial. If you think there are any errors, please let me know in the comments.\n","date":"May 18, 2011","externalUrl":null,"permalink":"/blog/2011/05/apphosted-com-django-hosting-review/","section":"Posts","summary":"","title":"AppHosted.com Django Hosting Service Review","type":"post"},{"content":"I work for a company called CashStar, most people have never heard of us, but a lot of people have used our products without even knowing it. CashStar is the man behind the curtain so to speak. We created an e-gift card platform that is used by some of the largest companies in the United States. Here is a short list of some of our customers: The Home Depot, Gap, Old Navy, Banana Republic, Staples, Coach, Starbucks, CVS, Chilis, Regal Cinemas, Pottery Barn, The CheeseCake Factory, Dell, Pizza Hut, and many more. To see the full list follow this link: http://www.cashstar.com/all-egift-card-clients/\nAnother thing that most people don\u0026rsquo;t know is that we power that platform using the awesome python framework called Django. Using Python and Django, our small engineering team in Portland, Maine has taken our startup which didn\u0026rsquo;t exist 3 years ago and turned it into a leader in the electronic gift card market. Due to this phenomenal growth, we need to expand our engineering team. We are currently looking for developers as well as QA folks to join our team. You will need to work on site in Portland Maine, and be legally allowed to work in the United States.\nIf you are interested in joining a fun, fast paced startup, please send the following to kencochrane@gmail.com with the subject line \u0026ldquo;CashStar.com Engineering Openings\u0026rdquo;. Failure to follow these simple directions will automatically disqualify you from these positions.\nResume Links to any projects you worked on recently Links to your github, bitbucket, stackoverflow, etc. accounts Availability (when you could start) How and when is best to contact you. More about CashStar.com: # All engineers receive a top of the line MacBook Pro and a 24\u0026quot; monitor. Our technology stack: : - Python - Django - Celery - MySQL - Linux - Mercurial - nginx - gunicorn - memcached - Redis - jQuery - virtualenv + pip - Selenium - Hudson We have a fully stocked kitchen with all kinds of snacks and all of the soda and coffee you can drink, and at least one free company lunch per week. We are located in the Old Port section of Portland Maine. If you have never visited Portland Maine before, you are missing out, it was ranked #1 on forbes.com\u0026rsquo;s America\u0026rsquo;s Most Livable Cities in 2009, and it was also ranked America\u0026rsquo;s Foodiest Small Town by Bon Appetit Magazine in 2009. Located right on the coast, and only a 90 minute drive to Boston, it is a great place to live and an even better place to raise a family. CashStar is privately held and has received venture funds from a multi-billion dollar global investment firm. If you have any questions feel free to contact me\ntwitter: @KenCochrane linkedin: http://www.linkedin.com/in/kencochrane ","date":"May 15, 2011","externalUrl":null,"permalink":"/blog/2011/05/django-job-opening-cashstar-portland-me/","section":"Posts","summary":"","title":"Django Job Opening @ CashStar.com (Portland, ME)","type":"post"},{"content":"","date":"May 15, 2011","externalUrl":null,"permalink":"/tags/jobs/","section":"Tags","summary":"","title":"Jobs","type":"tags"},{"content":"warning\nThis post is out of date, please read the new updated one:\nDeploying my Django application to dotCloud Part 2\nI have recently wrote about ep.io and gondor.io, two new python/django hosting services that are currently in beta. Another new service that has just recently hit the scene and is causing a lot of waves is DotCloud.com. DotCloud is a little different for a number of reasons. Their service isn\u0026rsquo;t targeting any one technology stack like ep.io and gondor.io with Python/Django/wsgi.\nDotCloud\u0026rsquo;s goal is to provide a bunch of different independent services that you can use as building blocks to build your application. If you need a database, take your pick from one of the many they support. Need an application that uses Django or Rails on the front end, and Java in the backend, that is fine, you can do that too. They realize that most developers don\u0026rsquo;t stick to one standard technology stack for all of their applications, so this allows you the flexibility to use the best tool for the job. It also gives you a nice playground to try out new services and see how they run without having to install, configure and maintain the service just for testing.\nDotCloud was part of the Y Combinator summer class of 2010, and they just recently raised $10 Million. With access to that much money, I\u0026rsquo;m guessing they will be hiring a bunch of people pretty quickly, and I\u0026rsquo;m sure we will be seeing their services expand as well.\nLike all new technologies, I want to play with them, and see if they live up to all of the hype. I was lucky enough to get a beta invite a few weeks ago, and I have been playing with it since. I decided that the best way to test it out would be to deploy my blog, and see how hard it would be. Here are the steps that I took to get my django based blog up and running on DotCloud. I ran into a few issues and I made sure that I put those in as well, so that if you get the same issue you can see how I solved it.\nDocumentation # Before I get started with any new service the first thing I usually do is look at the documentation. DotCloud has a nice list of documents along with some tutorials on how to get started. These 3 documents were the ones that I used the most.\nhttp://docs.dotcloud.com/tutorials/django/\nhttp://docs.dotcloud.com/components/mysql/\nhttp://docs.dotcloud.com/tutorials/firststeps/\nFirst Steps: # Like all cool services these days, dotcloud uses a python based CLI, so before we can get started we need to install the dotcloud client and configure it so that we can start using it.\n# create my dotcloud virtual environment. $ mkvirtualenv dotcloud # install dotcloud client using pip $ pip install dotcloud # create our application namespace called kencochrane $ dotcloud create kencochrane #enter api key that we got from: http://www.dotcloud.com/account/settings when prompted #\u0026lt;key goes here\u0026gt; Now that we have the client all setup, and an application namespace, now we can start building our service. I have forked my blog repository on bitbucket so that I could make dotcloud specific changes to it and not effect my original repo.\n# go into projects directory cd ~/projects # forked kencochranenet to kencochranenet_dotcloud, now clone that. locally hg clone https://bitbucket.org/kencochrane/kencochranenet_dotcloud kencochrane_dotcloud # go into the new directory. cd kencochrane_dotcloud Reading through the documentation tells me that I need to create a wsgi.py file and put in the root of my project. Using http://docs.dotcloud.com/tutorials/django/#djangowsgipy as a template, I created my wsgi.py file below. I had issues with the default template and I needed to add a directory to the sys.path so that wsgi could find my django apps correctly. Here is my finished file.\nimport os import sys # Ken added this, only thing that is different from the example template (not counting settings file name) sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__),\u0026#39;mysite\u0026#39;))) os.environ[\u0026#39;DJANGO_SETTINGS_MODULE\u0026#39;] = \u0026#39;mysite.settings\u0026#39; import django.core.handlers.wsgi djangoapplication = django.core.handlers.wsgi.WSGIHandler() def application(environ, start_response): if \u0026#39;SCRIPT_NAME\u0026#39; in environ: del environ[\u0026#39;SCRIPT_NAME\u0026#39;] return djangoapplication(environ, start_response) DotCloud uses PIP requirements files to manage your project dependencies. We already have our pip requirements file where it needs to be and named correctly so we don\u0026rsquo;t need to do anything, but if we didn\u0026rsquo;t have one, we would need to create one and put it in the root, and call it requirements.txt\nDotCloud requires that we put all of our static media in a directory called \u0026lsquo;static\u0026rsquo; in the root of our project. Because my static media is already found elsewhere, I need to create a directory called static and move all of my files into it.\n# we need to create a static directory to serve static media from (/static) . mkdir -p static If you use the django admin then you will need to create a postinstall script that will create a symlink under static for your django admin media files. Here is what mine looks like. If you use this as a template, don\u0026rsquo;t forget to change the DJANGO_SETTINGS_MODULE variable to match your project.\n#!/usr/bin/env python import os # To import anything under django.*, we must set this variable. os.environ[\u0026#39;DJANGO_SETTINGS_MODULE\u0026#39;] = \u0026#39;mysite.settings\u0026#39; # Import the admin module. The media directory is right under it! import django.contrib.admin # Retrieve the absolute path of the admin module. admindir = os.path.dirname(django.contrib.admin.__file__) # Add /media behind it. mediadir = os.path.join(admindir, \u0026#39;media\u0026#39;) # Compute the path of the symlink under the static directory. staticlink = os.path.join(\u0026#39;static\u0026#39;, \u0026#39;admin_media\u0026#39;) # If the link already exists, delete it. if os.path.islink(staticlink): os.unlink(staticlink) # Install the correct link. os.symlink(mediadir, staticlink) # make the script executable. chmod +x postinstall Because we changed our location of our static media, we need to remember to update our settings.py file with the new settings. Here are my changes.\n# edit settings.py file to the following. ADMIN_MEDIA_PREFIX = \u0026#39;/static/admin_media/\u0026#39; # change MEDIA_URL MEDIA_URL = \u0026#39;/static/\u0026#39; Deployment\nNow that we have our application\u0026rsquo;s project structure all setup and configured the way dotcloud wants it, we can start up some services and then deploy our app. If you want to find out what services they have available, you can run the following command.\n# find out which services that are available. dotcloud deploy -h You can chose among the following services: java host any Java servlet (also Clojure, Play!, and much more) mysql the worlds most popular open source database nodejs run JavaScript processes (including web apps) php host any PHP web app: Drupal, WordPress... php-worker run background PHP processes postgresql the worlds most advanced open source database python host any Python/WSGI web app: Django, Pylons, Web2py... python-worker run background Python processes rabbitmq AMQP message queue server redis advanced key-value store ruby host any Ruby/Rack web app: Rails, Sinatra... ruby-worker run background Ruby processes smtp authenticated SMTP relay to send e-mails reliably static host static HTTP content We need a database for our blog, since it was originally setup to use mysql, lets use that one here. Start up a new mysql service called \u0026lsquo;kencochrane.mysql\u0026rsquo;\n# start up the database service. $ dotcloud deploy -t mysql kencochrane.mysql Created \u0026#34;kencochrane.mysql\u0026#34;. Let\u0026rsquo;s make sure that it was created, and find out some more about our new database instance.\n# get the information about our new service $ dotcloud info kencochrane.mysqlcluster: wolverine config: mysql_password: password_was_changed created_at: 1303671517.96066 name: kencochrane.mysql namespace: kencochrane ports: - name: ssh url: ssh://dotcloud@mysql.kencochrane.dotcloud.com:3912 - name: mysql url: mysql://root:password_was_changed@mysql.kencochrane.dotcloud.com:3913 state: running type: mysql As you can see from above, we have a mysql database, it is running, and it lets us know the ssh and mysql urls and ports, along with the root password. Lets create our database. To do that we will login to the server and run some sql commands. (the sensitive information has been changed)\n# connect to dotcloud mysql server instance $ dotcloud run kencochrane.mysql -- mysql -u root -p # mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \\g. Your MySQL connection id is 34 Server version: 5.1.41-3ubuntu12.10 (Ubuntu) # create the user and database and give user permissions to database. Type \u0026#39;help;\u0026#39; or \u0026#39;\\h\u0026#39; for help. Type \u0026#39;\\c\u0026#39; to clear the current input statement. mysql\u0026gt; create database blogdb; Query OK, 1 row affected (0.00 sec) mysql\u0026gt; create user \u0026#39;blog_username\u0026#39; identified by \u0026#39;strong_password\u0026#39;; Query OK, 0 rows affected (0.05 sec) mysql\u0026gt; grant all on blogdb.* to \u0026#39;blog_user\u0026#39;@\u0026#39;%\u0026#39;; Query OK, 0 rows affected (0.04 sec) mysql\u0026gt; flush privileges; Query OK, 0 rows affected (0.00 sec) mysql\u0026gt; exit;Bye Shared connection to mysql.kencochrane.dotcloud.com closed. Now that we have our database all setup, we need to change our settings file so that it has all of the new information. Here is what mine looks like.\n# update your settings.py file. DATABASES = { \u0026#39;default\u0026#39;: { \u0026#39;ENGINE\u0026#39;: \u0026#39;django.db.backends.mysql\u0026#39;, \u0026#39;NAME\u0026#39;: \u0026#39;blogdb\u0026#39;, \u0026#39;USER\u0026#39;: \u0026#39;blog_username\u0026#39;, \u0026#39;PASSWORD\u0026#39;: \u0026#39;strong_password\u0026#39;, \u0026#39;HOST\u0026#39;: \u0026#39;mysql.kencochrane.dotcloud.com\u0026#39;, \u0026#39;PORT\u0026#39;: \u0026#39;3913\u0026#39;, } } Now we are ready to deploy our Django app, but before I go any further it is important to know the following. Dotcloud will pay attention to your .hgignore files. If you have a settings file in your .hgignore file so that it doesn\u0026rsquo;t get saved in the repo, it will not push those changes up to the cloud. You will need to remove it from the .hgignore in order to get those files out there. It is also import to remember that only changes that are committed are pushed, so don\u0026rsquo;t forget to commit your changes. If you wanted to be tricky you could use a post install script to pull down the file from a secure location and install it that way, if you want to make things super secure.\nNow that we have that behind us, lets get going. First we need to create a new python service.\n# deploy a new python app called kencochrane.blog $ dotcloud deploy --type python kencochrane.blog Service is now created so all we have to do is push our application to the new service.\n# push out your changes to the server $ dotcloud push kencochrane.blog . Now your code is out in the cloud, you just need to sync your database and run the migrations to get your database all setup, and you should be good to go.\n# sync the database $ dotcloud run kencochrane.blog python current/mysite/manage.py syncdb # we use south so run migrate $ dotcloud run kencochrane.blog python current/mysite/manage.py migrate now it works, just go to http://blog.kencochrane.dotcloud.com/ to see.\nIf you need to look at the logs to see how it is going you can do it two ways. The first way will tail your logs for you to your console.\n# look at logs of your service, it will tail them to your console. ctrl-c to stop. $ dotcloud logs kencochrane.blog Or login via ssh and look at your logs.\n# Open up a shell $ dotcloud ssh kencochrane.blog Here are the ones you most likely care about.\n# nginx access and error logs. /var/log/nginx/\u0026lt;app_name\u0026gt;.{access,error}.log # wsgi error logs /var/log/supervisor/uswgi.log If you need to restart your service just issue this command.\n# restart the service dotcloud restart kencochrane.blog DotCloud runs on Amazon\u0026rsquo;s EC2 service, and last week while I was doing my testing, I was getting a bunch of errors, I wasn\u0026rsquo;t sure if the errors were because this is a beta service, and they are still bugs in it, or if it was related to the amazon issues. Either way, I listed those errors below incase anyone else gets them. If I got the error I would just run the command again until it worked, sometimes it took 3 times before everything worked fine.\n# attempt 1 $ dotcloud push kencochrane.blog .DotCloud service unavailable (\u0026#34;No JSON object could be decoded\u0026#34;). Please try again later. If the problem persists, send an email to support@dotcloud.com. # attempt 2 $ dotcloud push kencochrane.blog . # upload . ssh://dotcloud@uploader.dotcloud.com:1060/kencochrane.blog # hg remote: Warning: Permanently added \u0026#39;[uploader.dotcloud.com]:1060,[174.129.15.77]:1060\u0026#39; (RSA) to the list of known hosts. pushing to ssh://dotcloud@uploader.dotcloud.com:1060/kencochrane.blog searching for changes remote: adding changesets remote: adding manifests remote: adding file changes remote: added 1 changesets with 3 changes to 3 files Scheduling build Fetching logs... Build started... mkdir: cannot create directory `126:afdea9036e83\u0026#39;: File exists Failed to fetch tarball (\u0026lt;type \u0026#39;exceptions.ValueError\u0026#39;\u0026gt;: Failed to extract the tarball, abort.) Build finished. Shared connection to blog.kencochrane.dotcloud.com closed. # 3rd time is a charm, it finally works. $ dotcloud push kencochrane.blog .# upload . ssh://dotcloud@uploader.dotcloud.com:1060/kencochrane.blog # hg remote: Warning: Permanently added \u0026#39;[uploader.dotcloud.com]:1060,[174.129.15.77]:1060\u0026#39; (RSA) to the list of known hosts. pushing to ssh://dotcloud@uploader.dotcloud.com:1060/kencochrane.blog searching for changes no changes found Scheduling build Fetching logs... Build started... . . all of the files (trimmed for space) . Fetched code revision 126:afdea9036e83 . . all the requirments (trimmed for space) . Cleaning up... Reloading nginx configuration: nginx. uwsgi: stopped uwsgi: started Build finished. Shared connection to blog.kencochrane.dotcloud.com closed. Conclusion: # Setting up my blog on DotCloud was fairly straightforward, the documentation helped guide me along the way. I did run into a few road blocks that stopped me in my tracks for a little while, but after some digging I was able to get past those issues. Hopefully this post will help you overcome the same issues I had, and make your deployment a little more smooth.\nThe service is very flexible, which is going to be good for most developers, but with the added flexibility they added some complexity that doesn\u0026rsquo;t exist on the other services. For example ep.io and gondor.io handle all database and user creation for you, so you don\u0026rsquo;t need to know what commands to run to create the database and the user, and they also automatically add the database connection strings to your settings file. I for one don\u0026rsquo;t mind the extra complexity considering I\u0026rsquo;m going to get the ability to have direct access to my database so that I can do what I want. Others, might not want to worry about that. If you are building an application for a client that isn\u0026rsquo;t very tech savy, the more things that are automated the better.\nDotCloud is still in beta, and if you take a look at their roadmap, they have very ambitious goals. I for one can\u0026rsquo;t wait to see how they progress. I have only gone into a few of the many things that dotcloud has to offer, I recommend that you try it out for yourself and leave me a comment letting me know how you liked it.\nNext time: # I have been lucky enough to get a sneak peak at http://apphosted.com a python/django hosting service similar to gondor.io and ep.io. I\u0026rsquo;m currently playing with the service now, once I\u0026rsquo;m finished I\u0026rsquo;ll post my findings.\nUpdate: # Read how this service stacks up against other services like it in my Django hosting roundup\n2/16/2012: Full disclosure. On Feb 16th 2012, I accepted a job with dotCloud. I plan on keeping this blog post up to date and impartial. If you think there are any errors, please let me know in the comments.\nwarning\nThis post is out of date, please read the new updated one:\nDeploying my Django application to dotCloud Part 2\n","date":"April 26, 2011","externalUrl":null,"permalink":"/blog/2011/04/deploying-my-django-application-to-dotcloud/","section":"Posts","summary":"","title":"Deploying my Django application to dotCloud","type":"post"},{"content":"The other day I wrote about a new Django hosting service called ep.io , after I made that post, I was lucky enough to get an invite for the private beta of gondor.io, which is similar to ep.io but from the folks over at Eldarion. In order to test out the service I decided to take my Dango blog application and deploy it to Gondor and I kept notes along the way, here are those notes.\nThe sign up process was pretty typical, register and then validate your email address. Once into the system you are offered the ability to create a site. So I created my site \u0026ldquo;KenCochrane_blog\u0026rdquo; and it gave me a key which I\u0026rsquo;ll use later on. Not knowing where to start I headed over to the documentation. Which, as of this writing, is pretty basic, which is understandable since they are still in beta. I\u0026rsquo;m assuming the documentation will get better as they move forward. The current documentation and this blog post should get you enough information to get going.\nSetting up my development environment: # Gondor uses a file to store your login credentials, so that you don\u0026rsquo;t need to type it in everytime. It is pretty basic, here is how I installed mine on my MacBook Pro (OSX 10.6).\n# create the file, and edit it. $ emacs ~/.gondor # copy this into the file. [auth] username = \u0026lt;my_username\u0026gt; password = \u0026lt;my_password\u0026gt; # change the permissions so that only you can access it. $ chmod 600 ~/.gondor Like all cool projects these days, gondor uses pip and virtualenv, I\u0026rsquo;m assuming you already have virtualenv and pip installed so I\u0026rsquo;ll move onto the fun stuff.\n# make my virtualenv called gondor $ mkvirtualenv gondor From this point on, all of these commands are done in the new gondor virtual environment that I just created. Gondor uses a command line client to manage your application, we need to install that before we can do anything.\n$ pip install gondor Since I\u0026rsquo;m going to need to make changes to my blog, I decided to fork it and make all of my changes to gondor on the new fork. The new fork is on bitbucket and it is called kencochranenet_gondor. Now that I have it forked I can pull it down locally.\n# go to my projects directory and clone my blog. $ cd ~/projects $ hg clone https://bitbucket.org/kencochrane/kencochranenet_gondor kencochranenet_gondor $ cd kencochranenet_gondor/mysite Now that we have some code to deploy, lets introduce the code to gondor. We use the init command inside of our django project and it will create a .gondor directory with a config file and some defaults.\n$ gondor init \u0026lt;site_key\u0026gt; Gondor expects your project to be in a certain layout in order to work. For more info see the documentation: https://gondor.io/support/project-layout/\nSince my blog didn\u0026rsquo;t have everything in the right place, I need to add some directories and move some files around. One of the files I needed to add was deploy/wsgi.py, here is what it contains. Notice the two sys.path.insert lines. I added those because the gondor documentation told me too.\nimport os, sys sys.path.insert(0, os.path.abspath(os.path.join(os.path.abspath(os.path.dirname(__file__)), os.pardir, os.pardir))) sys.path.insert(0, os.path.abspath(os.path.join(os.path.abspath(os.path.dirname(__file__)), os.pardir))) from django.core.handlers.wsgi import WSGIHandler os.environ[\u0026#34;DJANGO_SETTINGS_MODULE\u0026#34;] = \u0026#34;mysite.settings\u0026#34; application = WSGIHandler() I also had to add the same sys.path lines to my manage.py settings. Here is my new manage.py\n#!/usr/bin/env python from django.core.management import execute_manager import sys import os sys.path.insert(0, os.path.abspath(os.path.join(os.path.abspath(os.path.dirname(__file__)), os.pardir, os.pardir))) sys.path.insert(0, os.path.abspath(os.path.join(os.path.abspath(os.path.dirname(__file__)), os.pardir))) try: import settings # Assumed to be in the same directory. except ImportError: sys.stderr.write(\u0026#34;Error: Can\u0026#39;t find the file \u0026#39;settings.py\u0026#39; in the directory containing %r. It appears you\u0026#39;ve customized things.\\nYou\u0026#39;ll have to run django-admin.py, passing it your settings module.\\n(If the file settings.py does indeed exist, it\u0026#39;s causing an ImportError somehow.)\\n\u0026#34; % __file__) sys.exit(1) if __name__ == \u0026#34;__main__\u0026#34;: execute_manager(settings) Gondor will automagically add your database and other settings to your app for you at deployment, and in order for that to work correctly you need to add the following at the end of your settings.py file.\ntry: from local_settings import * except ImportError: pass My blog is using south to manage south migrations, so we need to edit the .gondor/config file to enable this.\n# edit .gondor/config changed migrations to migrations=south emacs .gondor/config # change this line migrations = None # to migrations = south Now that we have made all of the changes that we need. I need to commit all of my changes. It is important to note that because gondor uses mecurial or git for managing your project files, you need to always remember to add new files and commit all changes before you try to deploy your application.\nDeployment: # Next step is deployment. Before we get there, we need to create a new instance. Gondor allows you to deploy multiple versions of your code, this lets you have a production version and development versions all running at the same time. Gondor calls these different deployments \u0026lsquo;instances\u0026rsquo;. We run the \u0026lsquo;gondor create master\u0026rsquo; command, which will create a new instance called, you guessed it, \u0026lsquo;master\u0026rsquo;, and this is what you should see.\n$ gondor create master Reading configuration... [ok] Creating instance on Gondor... [ok] Run: gondor deploy master HEAD Visit: http://jg437.o1.gondor.io/ This created the master instance and tells us what our url for the instance will be. Now according to the log message I just need to run \u0026lsquo;gondor deploy master HEAD\u0026rsquo; and that will deploy my app. Lets do that now.\n$ gondor deploy master HEAD Reading configuration... [ok] ERROR: could not map \u0026#39;HEAD\u0026#39; to a SHA As you can see, it didn\u0026rsquo;t work, after lots of digging, I realized that the code was assuming I was on a branch called HEAD, which wasn\u0026rsquo;t correct. I think HEAD is the default git branch, and in mercirual, it is default. Just to be sure, I checked my mercurial branch.\n$ hg summary parent: 122:76f0c2271b7f tip gondor settings branch: default commit: (clean) update: (current) Running the quick \u0026lsquo;hg summary\u0026rsquo; command lets me know that my branch is default, so I then made the changes to my command and reran the script.\n$ gondor deploy master default Reading configuration... [ok] Building tarball from default... [ok] Pushing tarball to Gondor... Deploying... [failed] pip has failed installing your requirements. Here is the output we saw: Downloading/unpacking Django==1.2.5 (from -r mysite/requirements/project.txt (line 1)) Creating supposed download cache at /var/gondor/instances/.cache-i194/pip-download Storing download in cache at ./.cache-i194/pip-download/http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2FD%2FDjango%2FDjango-1.2.5.tar.gz Running setup.py egg_info for package Django Downloading/unpacking MySQL-python==1.2.3c1 (from -r mysite/requirements/project.txt (line 2)) Storing download in cache at ./.cache-i194/pip-download/http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2FM%2FMySQL-python%2FMySQL-python-1.2.3c1.tar.gz Running setup.py egg_info for package MySQL-python sh: mysql_config: not found Traceback (most recent call last): File \u0026#34;\u0026lt;string\u0026gt;\u0026#34;, line 14, in \u0026lt;module\u0026gt; File \u0026#34;/var/gondor/instances/.cache-i194/pip-build/MySQL-python/setup.py\u0026#34;, line 15, in \u0026lt;module\u0026gt; metadata, options = get_config() File \u0026#34;setup_posix.py\u0026#34;, line 43, in get_config libs = mysql_config(\u0026#34;libs_r\u0026#34;) File \u0026#34;setup_posix.py\u0026#34;, line 24, in mysql_config raise EnvironmentError(\u0026#34;%s not found\u0026#34; % (mysql_config.path,)) EnvironmentError: mysql_config not found Complete output from command python setup.py egg_info: sh: mysql_config: not found Traceback (most recent call last): File \u0026#34;\u0026lt;string\u0026gt;\u0026#34;, line 14, in \u0026lt;module\u0026gt; File \u0026#34;/var/gondor/instances/.cache-i194/pip-build/MySQL-python/setup.py\u0026#34;, line 15, in \u0026lt;module\u0026gt; metadata, options = get_config() File \u0026#34;setup_posix.py\u0026#34;, line 43, in get_config libs = mysql_config(\u0026#34;libs_r\u0026#34;) File \u0026#34;setup_posix.py\u0026#34;, line 24, in mysql_config raise EnvironmentError(\u0026#34;%s not found\u0026#34; % (mysql_config.path,)) EnvironmentError: mysql_config not found Command python setup.py egg_info failed with error code 1 We had mysql-python in our requirements file, and gondor doesn\u0026rsquo;t support mysql, so we need to remove that, commit and try again.\n$ gondor deploy master default Reading configuration... [ok] Building tarball from default... [ok] Pushing tarball to Gondor... Deploying... [failed] unable to build static (collectstatic or build_static not found) closer but no cigar, gondor uses staticfiles (new in django 1.3 or django-staticfiles) to manage static files. If you aren\u0026rsquo;t currently using static files add django-staticfiles to your pre django 1.3 app, or upgrade to django 1.3 and configure static files. To be honest, this was the biggest problem I had, it took me a while to get this correct. I was lucky enough to get some help on the #gondor IRC channel, which helped me along the way. With all of my changes made I committed them and tried again.\n$ gondor deploy master default Reading configuration... [ok] Building tarball from default... [ok] Pushing tarball to Gondor... Deploying... [ok] That worked!! woo hoo! now if I go to the URL (http://jg437.o1.gondor.io/) I got earlier, I should see something if all went well. I went to the site, and bingo, all working.. no data but it is there. Now how do I create my django admin user? Normally that is done when you do your initial syncdb, and since that is done by gondor, it doesn\u0026rsquo;t give you a chance. Looking thru all of the documentation I didn\u0026rsquo;t find anything. So I went and looked at the gondor client source code, and I found my answer. I kicked off the following command, answered the prompts and bingo, I was in business.\n$ gondor run master createsuperuser Conclusion # Now that I have my app up and running and I have admin access I could switch over my blog to the new service if I wanted. I\u0026rsquo;ll hold off for right now, but you get the picture. All and all I think this is going to be a nice service once they are fully up and running. It isn\u0026rsquo;t as far along as ep.io, but I\u0026rsquo;m sure it won\u0026rsquo;t be long before they are caught up. I\u0026rsquo;ll try and keep this post updated as I learn more about the service and they release new features.\nUpdate: # Read how this service stacks up against other services like it in my Django hosting roundup\n2/16/2012: Full disclosure. On Feb 16th 2012, I accepted a job with dotCloud a competitor to gondor.io. I plan on keeping this blog post up to date and impartial. If you think there are any errors, please let me know in the comments.\n","date":"April 22, 2011","externalUrl":null,"permalink":"/blog/2011/04/my-day-gondorio/","section":"Posts","summary":"","title":"My Day in Gondor.io","type":"post"},{"content":"Over the last decade I have deployed my applications many different ways, I have used, self managed dedicated servers, fully managed dedicated servers, Virtual Privates Servers from Linode, RackSpace and Amazon, shared hosting from webfaction, and I also used Google App Engine. I have learned a lot of things along the way, but the one thing I learned the most is that managing the server and keeping it up and running isn\u0026rsquo;t fun. The fun part is writing the applications, and watching them grow as people start to use them. If there was only a way to write the application and then let someone else manage all of the server stuff for you, for a reasonable price.\nThe Ruby on Rails folks have had services like this for a while, both Heroku and EngineYard are pretty popular and make hosting your Rails applications much easier. When Heroku was bought by salesforce.com for $212M, it opened people eyes, and started to get them thinking. Why can\u0026rsquo;t we do that for Django? I have been wanting to build a Django service like this for a while now, but with limited time and not knowing where to begin, my idea went no where. Recently there have been a bunch of new services that have popped up that provided turn key Django hosting, very similar in nature to Heroku, and Google App Engine (Python, and Java). This has gotten me very excited, and I plan on trying them all out.\nOne of these new services is called ep.io, and it is run by Andrew Godwin (of Django South fame) and Ben Firshman. They are trying to make the hosting as simple as possible, so that you don\u0026rsquo;t have to worry about servers at all. You write your application, deploy it to their service and they handle the rest. If you need to scale up your application, add load balancing, add more disk space, they will take care of that. This allows you the developer to spend your valuable time writing code instead of doing server maintenance. Best of all their pricing is based on usage, and they have a free tier, so if you have an application that you put out there, but it isn\u0026rsquo;t using any resources, or very few resources, it might not cost you anything at all. So there is nothing stopping you from trying it out.\nep.io is still in beta, and not available to the public yet. The only way to get access is to put your name on a waiting list, and wait until you are sent an invite. I was lucky enough to get an invite early on, and I have been playing around with ep.io for a new project that I\u0026rsquo;m working on. I\u0026rsquo;m still in development with my application so it isn\u0026rsquo;t public yet, but I have an app deployed and running. I personally feel that if you are lucky enough to be selected to a closed beta, that it is your duty to give as much feedback as possible, so that you can help make the application great. If you aren\u0026rsquo;t going to be doing this, then you don\u0026rsquo;t deserve the early access. That is why I\u0026rsquo;m writing this post, to help spread the word about ep.io, and to show people getting started with the service how easy it is to deploy their app.\nBefore you start using a service like ep.io, you need to do your research to make sure you know what it is you are getting with the service. Services like this aim to make hard tasks easy, and sometimes in doing so, they need to sacrifice features and flexibility. Knowing the pros and cons about a service before hand will help set your expectations accordingly so that you aren\u0026rsquo;t disappointed later on in the process. It is important to note that as of this writing, they are still in beta, and thus still a work in progress, so if you are reading this, make sure you look at their website for the most up to date information. I will try and come back and update this post as the service deploys new features, and my experience with the service grows.\nApplication setup:\nLike most managed application services, you won\u0026rsquo;t be able to deploy your application as is, you will need to make a few changes in order to get everything to work. Luckily for us, this step wasn\u0026rsquo;t too hard. I used the documentation that ep.io provides along with the ep.io project skeleton (https://github.com/idangazit/epio_skel) as a good starting point. ep.io uses an ini based configuration file called epio.ini for all of your configuration information. The documentation does a pretty good job explaining the different sections, and if you look at the ep.io project skeleton, there is a good example of one already created for you.\nep.io controls all aspects of your environment and because of this you don\u0026rsquo;t know what your database information is, but that is OK, because if you setup your django settings.py file correctly, they will handle all of that for you. Once again, checkout the skeleton project or the documentation for the recommended settings.\nDependencies:\nIf your project is already using virtualenv and pip, this shouldn\u0026rsquo;t be too difficult. If not, you will need to read up a little on pip and it\u0026rsquo;s requirements files (http://pip.openplans.org/requirement-format.html) in order to make sure you load all of your dependencies correctly. ep.io already has some of the most common libraries installed, for a full list, follow this link: http://www.ep.io/docs/runtime/#python-libraries. If you need a non pure-python library that isn\u0026rsquo;t on their list, just let them know and if it is reasonable, they will most likely install it for you.\nDeployment:\nep.io offers a simple control panel so that you can see what apps you have installed, how much they are costing you, how many resources they are using (database, bandwidth, CPU, disk usage, etc). It also allows you to configure your domains, set application permissions, and view the console log. Besides that, you need to do everything using their command line client. Before you can use the client, you will need to install it, you can do that using either pip or easy install.\npip install -U epio # or easy_install -U epio Once you have the client installed, this is where the fun begins. With the client you can create, suspend, resume, upload or delete your app. You can also use the client to kick off remote commands on your server such as django syncdb. See the official documentation for full details: http://www.ep.io/docs/client/\nHere are some notes and examples.\n#create app epio create [\u0026lt;app_name\u0026gt;] # suspend app epio suspend [-a \u0026lt;app_name\u0026gt;] #resume app epio resume [-a \u0026lt;app_name\u0026gt;] # delete app epio delete [-a \u0026lt;app_name\u0026gt;] # django syncdb epio django [-a \u0026lt;app_name\u0026gt;] syncdb # django run south migrations for all apps epio django [-a \u0026lt;app_name\u0026gt;] migrate # django run south migrations for just one app called chicken epio django [-a \u0026lt;app_name\u0026gt;] migrate chicken # bash command epio run_command [-a \u0026lt;app_name\u0026gt;] bash # psql access to your database. epio run_command [-a \u0026lt;app_name\u0026gt;] psql Uploading App:\nThe epio upload command will use git in the background to sync up your local directory to your server. It will ignore the .pyc files and such, if you have other files and directories that you also want to ignore you can create a file called .epioignore and list the files and directories to ignore in it. The .epioignore file has the same syntax as a .gitignore file.\nThe upload command is fine for most cases, but if you are already using git or mercurial, you can push your changes directly to ep.io, this allows you to do more of a continuous deployment setup, where you want to push to a central repo, run a bunch of tests and if it works push out to production. See this link for more info on uploading via git and mercurial http://www.ep.io/docs/vcss/\nDatabase:\nThe ep.io databases are behind a firewall for security purposes, and because of this, you don\u0026rsquo;t have direct access to the database, so if you are used to using a GUI database client, you are out of luck. The only way to get to your database right now is using the psql command tool.\n# psql access to your database. epio run_command [-a \u0026lt;app_name\u0026gt;] psql Getting data into your databases:\nThere are two recommended ways for getting data into your database.\nYou can create a SQL dump file, and add it to your project, and upload. Once it is up on the server you can run the following command.\nepio run_command -- psql -f dumpname.sql You can also stream the dump over the network via SSH. (They suggest that you only do this with dumps that are 20MB or less in size.)\nepio run_command psql \u0026lt; dumpname.sql Background tasks:\nep.io supports both cronjobs and background tasks via Celery and Redis.\nCron is pretty easy to setup, you have a section in the epio.ini file called cron where you put your normal cron commands. The syntax isn\u0026rsquo;t exactly the same, so see the documentation for the differences. http://www.ep.io/docs/epioini/#cron-section\nYou can use either plain celery or django-celery, what ever you want, they support most features, but they currently don\u0026rsquo;t support periodic tasks, they hope to get to that in the future. In the meantime just use a cron for those.\nep.io vs traditional hosting # I originally wrote this Pro\u0026rsquo;s and Con\u0026rsquo;s section at stackoverflow.com, and I include it here as well. http://stackoverflow.com/questions/5117395/django-hosting-on-ep-io/5648323#5648323\nPros: # Server Management: You don\u0026rsquo;t have to worry about your server setup at all, it handles everything for you. With a VPS, you would need to worry about making sure the server is up to date with security patches, and all that fun stuff, with this, you don\u0026rsquo;t worry about anything, they take care of all that for you. Deployment: It makes deploying an app and having it up and running really quickly. deploying a new version of an app is a piece of cake, I just need to run one maybe two commands, and it handles everything for me. Pricing: you are only charged for what you use, so if you have a very low traffic website, it might not cost you anything at all. Scaling: They handle scaling and load balancing for you out of the box, no need for you to worry about that. You still need to write your application so that it can scale efficiently, but if you do, they will handle the rest. Background tasks: They have support for cronjobs as well as background workers using celery. Customer support: I had a few questions, sent them an email, and had an answer really fast, they have been great, so much better then I would have expected. If you run your own VPS, you really don\u0026rsquo;t have anyone to talk to, so this is a major plus. Cons: # DB access: You don\u0026rsquo;t have direct access to the database, you can get to the psql shell, but you can\u0026rsquo;t connect an external client GUI. This makes doing somethings a little more difficult or slow. But you can still use the django admin or fixtures to do a lot of things. Limited services available: It currently only supports Postgresql and redis, so if you want to use MySQL, memcached, mongodb,etc you are out of luck. Low level c libs: You can\u0026rsquo;t install any dependencies that you want, similar to google app engine, they have some of the common c libs installed already, and if you want something different that isn\u0026rsquo;t already installed you will need to contact them to get it added. http://www.ep.io/docs/runtime/#python-libraries Email: You can\u0026rsquo;t send or receive email, which means you will need to depend on a 3rd party for that, which is probably good practice anyway, but it just means more money. File system: You have a more limited file system available to you, and because of the distributed nature of the system you will need to be very careful when working from files. You can\u0026rsquo;t (unless i missed it) connect to your account via (s)ftp to upload files, you will need to connect via the ep.io command line tool and either do an rsync or a push of a repo to get files up there. Update: # Read how this service stacks up against other services like it in my Django hosting roundup\n2/16/2012: Full disclosure. On Feb 16th 2012, I accepted a job with dotCloud a competitor to ep.io. I plan on keeping this blog post up to date and impartial. If you think there are any errors, please let me know in the comments.\n","date":"April 18, 2011","externalUrl":null,"permalink":"/blog/2011/04/my-experiences-with-epio/","section":"Posts","summary":"","title":"My Experiences with ep.io","type":"post"},{"content":"","date":"April 16, 2011","externalUrl":null,"permalink":"/tags/apache/","section":"Tags","summary":"","title":"Apache","type":"tags"},{"content":"","date":"April 16, 2011","externalUrl":null,"permalink":"/tags/macosx/","section":"Tags","summary":"","title":"Macosx","type":"tags"},{"content":"I always seem to forget this, so I\u0026rsquo;m going to post it here for easy reference.\nTo restart the Apache web server service you can run the following command in a Mac OS X Terminal window:\nsudo /usr/sbin/apachectl restart ","date":"April 16, 2011","externalUrl":null,"permalink":"/blog/2011/04/restart-apache-mac-os-x/","section":"Posts","summary":"","title":"Restart Apache in Mac OS X","type":"post"},{"content":"","date":"April 14, 2011","externalUrl":null,"permalink":"/tags/mysql/","section":"Tags","summary":"","title":"Mysql","type":"tags"},{"content":"","date":"April 14, 2011","externalUrl":null,"permalink":"/tags/percona/","section":"Tags","summary":"","title":"Percona","type":"tags"},{"content":"","date":"April 14, 2011","externalUrl":null,"permalink":"/tags/rhel5/","section":"Tags","summary":"","title":"RHEL5","type":"tags"},{"content":"","date":"April 14, 2011","externalUrl":null,"permalink":"/tags/upgrade/","section":"Tags","summary":"","title":"Upgrade","type":"tags"},{"content":"Upgrade the stock MySQL 5.0.x build to the new stable MySQL 5.1.x Percona version. Here are my notes on upgrading on Red Hat Enterprise Linux 5 (RHEL5) or CentOS5.\nFirst we need to stop mysql\nservice mysql stop; I\u0026rsquo;m assuming that you don\u0026rsquo;t have the percona repo installed on your machine, if not follow this link on how to set it up. Setting up Percona RPM Repo\nmkdir -p /tmp/downloads cd /tmp/downloads wget http://www.percona.com/redir/downloads/percona-release/percona-release-0.0-1.i386.rpm sudo rpm -Uhv percona-release-0.0-1.i386.rpm Now we need to remove the old MySQL 5.0 binaries. (Don\u0026rsquo;t worry this will leave your data.)\nyum remove -y mysql-server mysql mysql-devel mysql-test mysql-bench Now install the new Percona MySQL 5.1.x binaries. (depending on your connection this might take a few minutes)\nyum install -y Percona-Server-devel-51 Percona-Server-shared-51 Percona-Server-test-51 Percona-Server-client-51 Percona-Server-server-51 During the install it will throw out some warnings, make sure you write those down, and make the changes that it recommends. Here are a few of mine.\n110208 16:44:52 [Warning] \u0026#39;--log_slow_queries\u0026#39; is deprecated and will be removed in a future release. Please use \u0026#39;\u0026#39;--slow_query_log\u0026#39;/\u0026#39;--slow_query_log_file\u0026#39;\u0026#39; instead. 110208 16:44:52 [Warning] \u0026#39;--log\u0026#39; is deprecated and will be removed in a future release. Please use \u0026#39;\u0026#39;--general_log\u0026#39;/\u0026#39;--general_log_file\u0026#39;\u0026#39; instead. Assuming the install went well, now we need to upgrade the data to the new format. (replace \u0026lt;mysql admin user\u0026gt; with your mysql admin username, and enter password when prompted)\nmysql_upgrade -u \u0026lt;mysql admin username\u0026gt; -p Now if your upgrade was anything like mine it had a bunch of errors like this.\ndb_name.table_name error : Table upgrade required. Please do \u0026#34;REPAIR TABLE table_name\u0026#34; or dump/reload to fix it! When you try to repair the table you get this.\ndb_name.table_name note : The storage engine for the table doesn\u0026#39;t support repair Thanks to Peter Zaitsev mysql_upgrade and Innodb Tables we know that we can do the following to fix it.\nALTER TABLE db_name.table_name ENGINE=INNODB; We just need to run this command for each table that threw an error and it should fix it. (If you have lots of data this might take a while, I had some tables that took over 2 hours)\nOnce you think that you have all of the tables fixed, you can run this command and it will check all of your tables and let you know if you are good or not. If you still have errors, repeat the steps above until all tables pass.\nmysqlcheck -A --check-upgrade Hopefully everything passed for you, and you now have a fully upgraded database. It is important to note that there is another option besides running the table alter. You could have dumped all of the data prior to the upgraded and then restored it once the database was upgraded. This process would most likely take a lot longer.\n","date":"April 14, 2011","externalUrl":null,"permalink":"/blog/2011/04/upgrading-mysql-50x-percona-mysql-51x-rhel5-or-cen/","section":"Posts","summary":"","title":"Upgrading MySQL 5.0.x to Percona MySQL 5.1.x RHEL5 or CENTOS 5","type":"post"},{"content":"It includes the following:\nClass-based views Logging Extended static files handling unittest2 support Transaction context managers Configurable delete-cascade Contextual markers and comments for translatable strings Improvements to built-in template tags Caching changes For more information checkout the official release notes: http://docs.djangoproject.com/en/dev/releases/1.3/\n","date":"March 23, 2011","externalUrl":null,"permalink":"/blog/2011/03/django-13-has-been-released/","section":"Posts","summary":"","title":"Django 1.3 has been released!","type":"post"},{"content":"I have been using the Percona build of MySQL 5.0.x for a little while now, and I have been impressed. Now it is time to upgrade to the new stable MySQL 5.1.x version. Here are my notes on upgrading on Red Hat Enterprise Linux 5 (RHEL5) or CentOS5.\nFirst we need to stop mysql\nservice mysql stop; I\u0026rsquo;m assuming that you have the percona repo installed on your machine, if not see follow this link on how to set it up. Setting up Percona RPM Repo\nNow we need to remove the old MySQL 5.0 binaries. (Don\u0026rsquo;t worry this will leave your data.)\nyum remove -y Percona-SQL* Now install the new Percona MySQL 5.1.x binaries. (depending on your connection this might take a few minutes)\nyum install -y Percona-Server-devel-51 Percona-Server-shared-51 Percona-Server-test-51 Percona-Server-client-51 Percona-Server-server-51 During the install it will throw out some warnings, make sure you write those down, and make the changes that it recommends. Here are a few of mine.\n110208 16:44:52 [Warning] \u0026#39;--log_slow_queries\u0026#39; is deprecated and will be removed in a future release. Please use \u0026#39;\u0026#39;--slow_query_log\u0026#39;/\u0026#39;--slow_query_log_file\u0026#39;\u0026#39; instead. 110208 16:44:52 [Warning] \u0026#39;--log\u0026#39; is deprecated and will be removed in a future release. Please use \u0026#39;\u0026#39;--general_log\u0026#39;/\u0026#39;--general_log_file\u0026#39;\u0026#39; instead. Assuming the install went well, now we need to upgrade the data to the new format. (replace \u0026lt;mysql admin user\u0026gt; with your mysql admin username, and enter password when prompted)\nmysql_upgrade -u \u0026lt;mysql admin username\u0026gt; -p Now if your upgrade was anything like mine it had a bunch of errors like this.\ndb_name.table_name error : Table upgrade required. Please do \u0026#34;REPAIR TABLE table_name\u0026#34; or dump/reload to fix it! When you try to repair the table you get this.\ndb_name.table_name note : The storage engine for the table doesn\u0026#39;t support repair Thanks to Peter Zaitsev mysql_upgrade and Innodb Tables we know that we can do the following to fix it.\nALTER TABLE db_name.table_name ENGINE=INNODB; We just need to run this command for each table that threw an error and it should fix it. (If you have lots of data this might take a while, I had some tables that took over 2 hours)\nOnce you think that you have all of the tables fixed, you can run this command and it will check all of your tables and let you know if you are good or not. If you still have errors, repeat the steps above until all tables pass.\nmysqlcheck -A --check-upgrade Hopefully everything passed for you, and you now have a fully upgraded database. It is important to note that there is another option besides running the table alter. You could have dumped all of the data prior to the upgraded and then restored it once the database was upgraded. This process would most likely take a lot longer.\n","date":"February 15, 2011","externalUrl":null,"permalink":"/blog/2011/02/upgrading-percona-mysql-50x-percona-mysql-51x-rhel/","section":"Posts","summary":"","title":"Upgrading Percona MySQL 5.0.x to Percona MySQL 5.1.x RHEL5 or CENTOS 5","type":"post"},{"content":" Django 1.2 has been released # I have been waiting for this release for a while now, mostly because it includes some of the features I have been dying to get my hands on. The ones I like the most are\nsupport for multiple databases smart-if template tag The new Email Backends relaxed requirements for usernames. Here is a list for some of the features, and a little bit about each one.\nSupport for multiple databases # Django 1.2 adds the ability to use more than one database in your Django project. Queries can be issued at a specific database with the using() method on QuerySet objects. Individual objects can be saved to a specific database by providing a using argument when you call save().\nModel validation # Model instances now have support for validating their own data, and both model and form fields now accept configurable lists of validators specifying reusable, encapsulated validation behavior. Note, however, that validation must still be performed explicitly. Simply invoking a model instance’s save() method will not perform any validation of the instance’s data.\nImproved CSRF protection # Django now has much improved protection against Cross-Site Request Forgery (CSRF) attacks. This type of attack occurs when a malicious Web site contains a link, a form button or some JavaScript that is intended to perform some action on your Web site, using the credentials of a logged-in user who visits the malicious site in their browser. A related type of attack, “login CSRF,” where an attacking site tricks a user’s browser into logging into a site with someone else’s credentials, is also covered.\nMessages framework # Django now includes a robust and configurable messages framework with built-in support for cookie- and session-based messaging, for both anonymous and authenticated clients. The messages framework replaces the deprecated user message API and allows you to temporarily store messages in one request and retrieve them for display in a subsequent request (usually the next one).\nObject-level permissions # A foundation for specifying permissions at the per-object level has been added. Although there is no implementation of this in core, a custom authentication backend can provide this implementation and it will be used by django.contrib.auth.models.User. See the authentication docs for more information.\nPermissions for anonymous users # If you provide a custom auth backend with supports_anonymous_user set to True, AnonymousUser will check the backend for permissions, just like User already did. This is useful for centralizing permission handling - apps can always delegate the question of whether something is allowed or not to the authorization/authentication backend. See the authentication docs for more details.\nRelaxed requirements for usernames # The built-in User model’s username field now allows a wider range of characters, including @, +, . and - characters.\nE-mail backends # You can now configure the way that Django sends e-mail. Instead of using SMTP to send all e-mail, you can now choose a configurable e-mail backend to send messages. If your hosting provider uses a sandbox or some other non-SMTP technique for sending mail, you can now construct an e-mail backend that will allow Django’s standard mail sending methods to use those facilities. This also makes it easier to debug mail sending. Django ships with backend implementations that allow you to send e-mail to a file, to the console, or to memory. You can even configure all e-mail to be thrown away.\n“Smart” if tag # The if tag has been upgraded to be much more powerful. First, we’ve added support for comparison operators. The operators supported are ==, !=, \u0026lt;, \u0026gt;, \u0026lt;=, \u0026gt;=, in and not in, all of which work like the Python operators, in addition to and, or and not, which were already supported.\nTemplate caching # In previous versions of Django, every time you rendered a template, it would be reloaded from disk. In Django 1.2, you can use a cached template loader to load templates once, then cache the result for every subsequent render. This can lead to a significant performance improvement if your templates are broken into lots of smaller subtemplates (using the {% extends %} or {% include %} tags). As a side effect, it is now much easier to support non-Django template languages. For more details, see the notes on supporting non-Django template languages.\nNatural keys in fixtures # Fixtures can now refer to remote objects using Natural keys. This lookup scheme is an alternative to the normal primary-key based object references in a fixture, improving readability and resolving problems referring to objects whose primary key value may not be predictable or known.\nFast failure for tests # Both the test subcommand of django-admin.py and the runtests.py script used to run Django\u0026rsquo;s own test suite now support a \u0026ndash;failfast option. When specified, this option causes the test runner to exit after encountering a failure instead of continuing with the test run. In addition, the handling of Ctrl-C during a test run has been improved to trigger a graceful exit from the test run that reports details of the tests that were run before the interruption.\nBigIntegerField # Models can now use a 64-bit BigIntegerField type.\nImproved localization # Django\u0026rsquo;s internationalization framework has been expanded with locale-aware formatting and form processing. That means, if enabled, dates and numbers on templates will be displayed using the format specified for the current locale. Django will also use localized formats when parsing data in forms. See Format localization for more details.\nreadonly_fields in ModelAdmin # django.contrib.admin.ModelAdmin.readonly_fields has been added to enable non-editable fields in add/change pages for models and inlines. Field and calculated values can be displayed alongside editable fields.\nAnd much much more!! # To find out what else is in this release visit Django 1.2 offical release notes. Download it here: http://www.djangoproject.com/download/1.2/tarball/ Offical website: DjangoProject.com. ","date":"May 17, 2010","externalUrl":null,"permalink":"/blog/2010/05/django-12-has-been-released/","section":"Posts","summary":"","title":"Django 1.2 has been Released!","type":"post"},{"content":"I was curious so I went ahead and found out all of the official Django. releases and when they were released and how long it took between each release.\nThere have been 7 major releases and 8 minor releases in the past 5 years. I wonder how this compares to other projects like Ruby on Rails.\nRelease # Date # of months since last release\nOpen Sourced Jul 15, 2005 n/a 0.90 Nov 16, 2005 4 0.91 Jan 11, 2006 2 0.95 Jul 29, 2006 6 0.95.1 Jan 21, 2007 6 0.96 Mar 23, 2007 2 0.96.1 Oct 26, 2007 7 0.96.2 May 14, 2008 7 1.0 Sep 3, 2008 4 1.0.1 Nov 15, 2008 2 1.0.2 Nov 18, 2008 0 1.1 Jul 29, 2009 8 1.1.1 Oct 9, 2009 3 1.1.2 May 17, 2010 7 1.1.3 Dec 22, 2010 7 1.1.4 Feb 8, 2011 2 1.2 May 17, 2010 7 1.2.1 May 18, 2010 0 1.2.2 Sep 8, 2010 4 1.2.3 Sep 10, 2010 0 1.2.4 Dec 22, 2010 3 1.2.5 Feb 8, 2011 2 1.2.6 Sep 9, 2011 7 1.2.7 Sep 10, 2011 0 1.3 Mar 23, 2011 10 1.3.1 Sep 9, 2011 6 1.4 Mar 23, 2012 6\n","date":"May 17, 2010","externalUrl":null,"permalink":"/blog/2010/05/django-project-release-history/","section":"Posts","summary":"","title":"Django Project Release History","type":"post"},{"content":"I created this script that will download memcached build,install and set it up to start up automatically on reboot. It also installs the init.d script to manage memcache so you can restart etc.\nYou can either copy and paste it into your shell prompt using a user that has install privileges (root,admin,etc) or you can copy into a .sh file and execute it.\nThe code for most of this was take from this blog post: http://www.vbseo.com/blogs/danny-bembibre/daemon-scripts-memcached-44/ it doesn\u0026rsquo;t seem to be around anymore so I put it here for anyone else looking for this very helpful info.\nwget http://memcached.googlecode.com/files/memcached-1.4.5.tar.gz tar -xvzf memcached-1.4.5.tar.gz cd memcached-1.4.5 ./configure make make test make install touch /etc/memcached.conf cat \u0026lt;\u0026lt; EOF \u0026gt;\u0026gt; /etc/memcached.conf #Memory a user -m 64 # default port -p 11211 # user to run daemon nobody/apache/www-data -u nobody # only listen locally -l 127.0.0.1 EOF touch /etc/init.d/memcached chmod +x /etc/init.d/memcached cat \u0026lt;\u0026lt; EOF \u0026gt;\u0026gt; /etc/init.d/memcached #!/bin/bash # # memcached This shell script takes care of starting and stopping # standalone memcached. # # chkconfig: - 80 12 # description: memcached is a high-performance, distributed memory # object caching system, generic in nature, but # intended for use in speeding up dynamic web # applications by alleviating database load. # processname: memcached # config: /etc/memcached.conf # Source function library. . /etc/rc.d/init.d/functions PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin DAEMON=/usr/local/bin/memcached DAEMONBOOTSTRAP=/usr/local/bin/start-memcached DAEMONCONF=/etc/memcached.conf NAME=memcached DESC=memcached PIDFILE=/var/run/$NAME.pid [ -x $DAEMON ] || exit 0 [ -x $DAEMONBOOTSTRAP ] || exit 0 RETVAL=0 start() { echo -n $\u0026#34;Starting $DESC: \u0026#34; daemon $DAEMONBOOTSTRAP $DAEMONCONF RETVAL=$? [ $RETVAL -eq 0 ] \u0026amp;\u0026amp; touch $PIDFILE echo return $RETVAL } stop() { echo -n $\u0026#34;Shutting down $DESC: \u0026#34; killproc $NAME RETVAL=$? echo [ $RETVAL -eq 0 ] \u0026amp;\u0026amp; rm -f $PIDFILE return $RETVAL } # See how we were called. case \u0026#34;$1\u0026#34; in start) start ;; stop) stop ;; restart|reload) stop start RETVAL=$? ;; status) status $prog RETVAL=$? ;; *) echo $\u0026#34;Usage: $0 {start|stop|restart|status}\u0026#34; exit 1 esac exit $RETVAL EOF touch /usr/local/bin/start-memcached chmod +x /usr/local/bin/start-memcached cat \u0026lt;\u0026lt; EOF \u0026gt;\u0026gt; /usr/local/bin/start-memcached #!/usr/bin/perl -w # start-memcached # 2003/2004 - Jay Bonci \u0026lt;jaybonci@debian.org\u0026gt; # This script handles the parsing of the /etc/memcached.conf file # and was originally created for the Debian distribution. # Anyone may use this little script under the same terms as # memcached itself. use strict; if ($\u0026gt; != 0 and $\u0026lt; != 0) { print STDERR \u0026#34;Only root wants to run start-memcached.\\n\u0026#34;; exit; } my $etcfile = shift || \u0026#34;/etc/memcached.conf\u0026#34;; my $params = []; my $etchandle; # This script assumes that memcached is located at /usr/bin/memcached, and # that the pidfile is writable at /var/run/memcached.pid my $memcached = \u0026#34;/usr/local/bin/memcached\u0026#34;; my $pidfile = \u0026#34;/var/run/memcached.pid\u0026#34;; # If we don\u0026#39;t get a valid logfile parameter in the /etc/memcached.conf file, # we\u0026#39;ll just throw away all of our in-daemon output. We need to re-tie it so # that non-bash shells will not hang on logout. Thanks to Michael Renner for # the tip my $fd_reopened = \u0026#34;/dev/null\u0026#34;; sub handle_logfile { my ($logfile) = @_; $fd_reopened = $logfile; } sub reopen_logfile { my ($logfile) = @_; open *STDERR, \u0026#34;\u0026gt;\u0026gt;$logfile\u0026#34;; open *STDOUT, \u0026#34;\u0026gt;\u0026gt;$logfile\u0026#34;; open *STDIN, \u0026#34;\u0026gt;\u0026gt;/dev/null\u0026#34;; $fd_reopened = $logfile; } # This is set up in place here to support other non -[a-z] directives my $conf_directives = { \u0026#34;logfile\u0026#34; =\u0026gt; \\\u0026amp;handle_logfile }; if (open $etchandle, $etcfile) { foreach my $line (\u0026lt;$etchandle\u0026gt;) { $line =~ s/\\#.*//go; $line = join \u0026#39; \u0026#39;, split \u0026#39; \u0026#39;, $line; next unless $line; next if $line =~ /^\\-[dh]/o; if ($line =~ /^[^\\-]/o) { my ($directive, $arg) = $line =~ /^(.*?)\\s+(.*)/; $conf_directives-\u0026gt;{$directive}-\u0026gt;($arg); next; } push @$params, $line; } } unshift @$params, \u0026#34;-u root\u0026#34; unless (grep $_ eq \u0026#39;-u\u0026#39;, @$params); $params = join \u0026#34; \u0026#34;, @$params; if (-e $pidfile) { open PIDHANDLE, \u0026#34;$pidfile\u0026#34;; my $localpid = \u0026lt;PIDHANDLE\u0026gt;; close PIDHANDLE; chomp $localpid; if (-d \u0026#34;/proc/$localpid\u0026#34;) { print STDERR \u0026#34;memcached is already running.\\n\u0026#34;; exit; } else { `rm -f $localpid`; } } my $pid = fork(); if ($pid == 0) { reopen_logfile($fd_reopened); exec \u0026#34;$memcached $params\u0026#34;; exit(0); } elsif (open PIDHANDLE,\u0026#34;\u0026gt;$pidfile\u0026#34;) { print PIDHANDLE $pid; close PIDHANDLE; } else { print STDERR \u0026#34;Can\u0026#39;t write pidfile to $pidfile.\\n\u0026#34;; } EOF /etc/init.d/memcached restart /sbin/chkconfig memcached on ","date":"May 14, 2010","externalUrl":null,"permalink":"/blog/2010/05/automated-install-script-for-memcached-on-rhel5/","section":"Posts","summary":"","title":"Automated Install Script for memcached on RHEL5","type":"post"},{"content":"","date":"May 14, 2010","externalUrl":null,"permalink":"/tags/install/","section":"Tags","summary":"","title":"Install","type":"tags"},{"content":"","date":"May 14, 2010","externalUrl":null,"permalink":"/tags/memcached/","section":"Tags","summary":"","title":"Memcached","type":"tags"},{"content":"","date":"May 14, 2010","externalUrl":null,"permalink":"/tags/script/","section":"Tags","summary":"","title":"Script","type":"tags"},{"content":" Installing Mercurial 1.5.1 on Red Hat Enterprise Linux 5 (RHEL5) using Python 2.5 # My RHEL5 setup uses python2.5 which I compiled from source, since RHEL5 only comes with python2.4.\nBecause I didn\u0026rsquo;t use the built in python 2.4 version I couldn\u0026rsquo;t do the simple\n$ yum install mercurial Instead I need to build mercurial from source, which isn\u0026rsquo;t too bad. Just do the following.\nas Root\n$ wget http://mercurial.selenic.com/release/mercurial-1.5.1.tar.gz $ tar -xvzf mercurial-1.5.1.tar.gz $ cd mercurial-1.5.1 $ make PYTHON=python2.5 install If you get the following error:\n$ python2.5 setup.py build $ Couldn\u0026#39;t import standard bz2 (incomplete Python install). make: [build] Error 1 Then when you installed python 2.5 you didn\u0026rsquo;t have the bzip2 libs installed so it didn\u0026rsquo;t include them in the python2.5 install. No big deal all you need to do is install those libs and then recomplile python 2.5\ninstall bzip2 libs\n$ yum install bzip2 bzip2-devel bzip2-libs Download 2.5.4 from python.org\n$ wget http://python.org/ftp/python/2.5.4/Python-2.5.4.tar.bz2 $ tar -xvjf Python-2.5.4.tar.bz2 $ cd Python-2.5.4 switch to root:\n$ ./configure $ make $ make altinstall Once complete (this will take a few minutes) rerun the steps above and then you should be good.\ngo back to where you untared the mercurial bundle\n$ cd mercurial-1.5.1 $ make PYTHON=python2.5 install Check to make sure you are running 1.5.1\n$ hg --version Mercurial Distributed SCM (version 1.5.1) That is all, now you are running Mercurial 1.5.1 on RHEL5 with python2.5!\n","date":"May 13, 2010","externalUrl":null,"permalink":"/blog/2010/05/mercurial-1_5_1-on-rhel5-using-python2_5/","section":"Posts","summary":"","title":"Installing Mercurial 1.5.1 on Red Hat Enterprise Linux 5 (RHEL5) using Python 2.5","type":"post"},{"content":"","date":"May 13, 2010","externalUrl":null,"permalink":"/tags/mercurial/","section":"Tags","summary":"","title":"Mercurial","type":"tags"},{"content":"","date":"January 18, 2010","externalUrl":null,"permalink":"/tags/apple/","section":"Tags","summary":"","title":"Apple","type":"tags"},{"content":"","date":"January 18, 2010","externalUrl":null,"permalink":"/tags/application/","section":"Tags","summary":"","title":"Application","type":"tags"},{"content":"","date":"January 18, 2010","externalUrl":null,"permalink":"/tags/face-flip/","section":"Tags","summary":"","title":"Face Flip","type":"tags"},{"content":"I have finished with my latest iPhone application and I have submitted it to Apple. I submitted it on Sunday 01/17/2010, and hopefully it will be approved by the end of the month. Normally it takes about 10-14 days for apple to approve apps but there have been recent reports that apps have been approved in as little as 2-3 days. I\u0026rsquo;ll keep my fingers crossed and post when I hear back.\nFace Flip is a simple iPhone application that allows you to take your photos and mix and match them with other photos to make crazy faces. You can add photos to the app using the built in iPhone Camera or it will take photos from the iPhone photo album. Once the photos are in the app you can switch to mixer mode and the app will slice the photos into 3 separate parts, Top, middle and bottom. On the mixer mode it will allow you to swipe the different parts, left or right, until you get to a cool combo. Once you find the perfect Face Flip photo you can export the photos from the app, you can either email them to your friends, or save it to iPhone photo album, or both.\nThe app was built to be very simple, and easy to use, and most of all FUN.\nIf you get a chance, go ahead and download it, and let me know what you think.\nHere are some screen shots.\n","date":"January 18, 2010","externalUrl":null,"permalink":"/blog/2010/01/introducing-my-new-iphone-application-face-flip/","section":"Posts","summary":"","title":"Introducing my new iPhone application Face Flip","type":"post"},{"content":"","date":"January 18, 2010","externalUrl":null,"permalink":"/tags/iphone/","section":"Tags","summary":"","title":"IPhone","type":"tags"},{"content":"Recently I decided it was time to upgrade my aging 36\u0026quot; CRT television with a new HDTV. After doing a lot of research I narrowed down my requirements to the following:\nAt least 42\u0026quot;HDTVLow power consumptionAbility to hook up my Mac Mini so that I can watch internet TV (hulu, netflix, etc).LCD Flat panelDigital TV tuneUnder $800.00I looked around for a few months and finally ended up with a Vizio 42\u0026quot; Class Eco 1080p 60Hz LCD HDTV, model # VO420E , that I bought at walmart for $648.00 . I really like this TV, it has everything I wanted, and the price was decent.\nI have had the TV for a few weeks now, and the only thing that really annoys me is the fact that my non HD channels are separate from my HD channels. My non HD channels are 1 thru 70, and my HD channels are in this weird format, 6-0,83-1,83-2,84-11, etc. Here is what NBC,CBS,ABC, and Fox look like. (It is important to note, that I might be off with the HD channel numbers, since I can\u0026rsquo;t seem to remember which channel is which.)\nNBC: [non HD channel , 6] , [HD channel , 6-0].\nCBS: [non HD channel , 13] , [HD channel , 83-11].\nABC: [non HD channel , 8] , [HD channel , 84-2].\nFOX: [non HD channel , 7] , [HD channel , 82-4].\nNow that I have an HDTV, I have become somewhat of an HD snob, why would I want to watch something that isn\u0026rsquo;t HD if it was available. So I end up channel surfing thru the channels, and then if it ends up on a channel where I have HD, I then search for the HD channel. For NBC it was kind of easy to fix this because when you surf the channels it would put 6 and 6-0 right next to each other, so I ended up removing channel 6 all together, so now it goes 4,5,6-0,7,8,etc.\nFeature Request:\nIf there are any TV manufactures out there, here is what I would like you to add to your next TV. I would like the ability to change my channel order when I am surfing. If I only watch 20 channels, why should I have to scan through all 70+ channels finding the few that I want, why not allow me to reorder them so that I can put my favorite 20 first, and then put the rest of the channels after. This way I could replace any non HD channels with their HD equivalents and I wouldn\u0026rsquo;t need to worry about searching for those every time. If you would like bonus points, it would be great if you could have a custom list for each person in the house, so that my wife can have her list of channels, and I can have mine. I envision it working like this. I pick up my remote, I hit my special button (there are 4 buttons, one for each profile), this tells the TV which channel ordering to use, and then when I start browsing, which channels to show next. My wife sits down, hits her button, and then the remote knows it is her, and when she starts browsing, she gets the channels she wants, in her order. We have favorite channel buttons, when extend that a little bit, and make it much more useful.\nThis feature seems very reasonable, and technology wise, not too hard. I mean come on, they just released 3D TV, that has to be much harder then this.\nWell that is just my 2 cents, if you know of a TV or set top box that has this feature please let me know.\n","date":"January 11, 2010","externalUrl":null,"permalink":"/blog/2010/01/new-tv-feature-request-ability-to-set-the-channel-order/","section":"Posts","summary":"","title":"New TV Feature Request: Ability to set the channel order","type":"post"},{"content":"","date":"March 9, 2009","externalUrl":null,"permalink":"/tags/django-notification/","section":"Tags","summary":"","title":"Django-Notification","type":"tags"},{"content":"I was setting up Django-Notification for a project, and I was trying to create notice types like it mentions on their wiki . Their wiki says that you should add the following code to your management.py file in your app.\nfrom django.conf import settings\nfrom django.db.models import signals\nfrom django.utils.translation import ugettext_noop as _\nif \u0026ldquo;notification\u0026rdquo; in settings.INSTALLED_APPS:\nfrom notification import models as notification\ndef create_notice_types(app, created_models, verbosity, **kwargs):\nnotification.create_notice_type(\u0026ldquo;friends_invite\u0026rdquo;, _(\u0026ldquo;Invitation Received\u0026rdquo;), _(\u0026ldquo;you have received an invitation\u0026rdquo;))\nnotification.create_notice_type(\u0026ldquo;friends_accept\u0026rdquo;, _(\u0026ldquo;Acceptance Received\u0026rdquo;), (\u0026ldquo;an invitation you sent has been accepted\u0026rdquo;))\nsignals.post_syncdb.connect(create_notice_types, sender=notification)\nelse:\nprint \u0026ldquo;Skipping creation of NoticeTypes as notification app not found\u0026rdquo;\nAfter trying lots of different things (dropping database and re syncing,etc) the management file wasn\u0026rsquo;t getting picked up and it was quite annoying.\nIt turns out that if you have written custom management extensions and you have a management directory under your application it will ignore the management.py file all together. So you need to add the code to the app/management/init.py file instead.\nOnce you have added the files you need to resync (make sure you remove notification* tables first).\nThis makes total sense now, and I guess I shouldn\u0026rsquo;t have taken the directions to literaly.\nIf you still can\u0026rsquo;t get it to work, the other option is to just create a fixture and load the data that way.\n","date":"March 9, 2009","externalUrl":null,"permalink":"/blog/2009/03/django-notification-creating-notice-types-issues/","section":"Posts","summary":"","title":"Django-Notification Creating Notice Types issues","type":"post"},{"content":"","date":"March 9, 2009","externalUrl":null,"permalink":"/tags/management.py/","section":"Tags","summary":"","title":"Management.py","type":"tags"},{"content":"","date":"February 26, 2009","externalUrl":null,"permalink":"/tags/cgi-bin/","section":"Tags","summary":"","title":"Cgi-Bin","type":"tags"},{"content":"","date":"February 26, 2009","externalUrl":null,"permalink":"/tags/plesk/","section":"Tags","summary":"","title":"Plesk","type":"tags"},{"content":"I recently worked on a project to quickly create a simple cgi-bin script to capture some data from a form on a mostly static website. The script was simple, take the data from the form, and send it via email. The script was written by a co-worker of mine using python. When we went to deploy the script on the server with Plesk 8.6, it was nothing but a pain in the butt.. I have documented some of the things we did below in order to get it working.\nHere are the errors that we were getting during the process in no particular order.\n\u0026ldquo;target uid/gid (10001/2524 or 2523) mismatch with directory (10001/2524) or program (48/48)\u0026quot;\u0026ldquo;uid: (10001/cashstarftp) gid: (2524/2524) cmd: mail_script.cgi\u0026rdquo;\u0026ldquo;cannot get docroot information (/var/www/vhosts)\u0026quot;\u0026ldquo;failed to open log file /var/log/httpd/suexec_log\u0026rdquo;\u0026ldquo;Premature end of script headers: mail_script.py\u0026rdquo;\nHere are the different steps that we did to resolve the different errors:\nSSH in and type the following: \u0026ldquo;chown root:apache /usr/sbin/suexec\u0026rdquo;\nMy permissions were also wrong for the following\n/var/log/httpd, should be drwx\u0026mdash;\u0026mdash; root root\n/var/log/httpd/suexec_log, should be -rw-r\u0026ndash;r\u0026ndash; root root\n/home/httpd/vhosts/your-domain-name.com/statistics, should be dr-xr-x\u0026mdash; root psaserv\n/home/httpd/vhosts/your-domain-name.com/cgi-bin should be, drwxr-x\u0026ndash;x ftpuser psaserv\nAs most of you know but just in case scripts in the cgi-bin directory need to be -rwxr-xr-x ftpuser psacln\nReplace \u0026ldquo;ftpuser\u0026rdquo; with the ftp user name for that domain\nAnd don\u0026rsquo;t forget to restart Apache after making these changes.\nThere are probably more things that we ended up trying that I didn\u0026rsquo;t document here, so feel free to post how you fixed your issue, so that others can benefit.\n","date":"February 26, 2009","externalUrl":null,"permalink":"/blog/2009/02/plesk-8.6-cgi-bin-trouble/","section":"Posts","summary":"","title":"Plesk 8.6 cgi-bin trouble","type":"post"},{"content":"A little while ago, I noticed that Google started hosting the most common JavaScript libraries on their CDN. The project is called AJAX Libraries API , and it allows websites to reference these libaries instead of hosting the files themselves. Using a CDN to host your static files isn\u0026rsquo;t anything new, and it provides some nice side effects.\nLess bandwidthLess space (every little bit helps)Global cachingIf everyone uses the same file it is more likely they already have the file cached from another site.\nFaster download\nClients get the file from the server closer to themDifferent servers mean more asynchronous downloadsFor more info check out the YSlow page.\nAt first this sounded great, Google was providing a nice service for free, and everyone benefits. But then I was thinking, what does Google get out of this? With a big company there is always something, even Google.\nThen it hit me. How does Google Analytics work? If you don\u0026rsquo;t know, it is pretty simple. All you need to do is insert a small little JavaScript snippet at the bottom of the pages you want to track, and that is it. How is Google analytics different then the new Google Ajax API? Not much, the major difference is that Analytics has a tracking code and the AJAX API doesn\u0026rsquo;t. But that isn\u0026rsquo;t a big deal for Google to get around, they can easily link all views back to the originatly server. So the tracking code isn\u0026rsquo;t needed unless you want to link that back to a user, so that they can view the results.\nAJAX API, isn\u0026rsquo;t the only project where they could easily gather information. They also have Google Chart API where they will generate charts for you on the file to display on your website. The big one that most people see everyday is the Google Maps API which allow you to embed google maps onto any website.\nIf you combine the information they get from all the people visiting thier website, with the information they gather from Google Analytics and all of these free tools that they give away, it is really amazing how much data they have, and how easy it would be to track people.\nI\u0026rsquo;m not a paranoid person, and I could care less if Google wanted to track me, but I can see how some people might be concerned. If they could only use all of this information to track down all of the bad people out there. I don\u0026rsquo;t know if Osama bin laden even has power in his cave, no less internet access, but if he did, watch out. Google\u0026rsquo;s Coming!\n","date":"February 4, 2009","externalUrl":null,"permalink":"/blog/2009/02/googles-ajax-libary-hosting-the-new-web-tracker/","section":"Posts","summary":"","title":"Google's Ajax Libary hosting, the New Web Tracker","type":"post"},{"content":"","date":"February 1, 2009","externalUrl":null,"permalink":"/tags/100boxes.com/","section":"Tags","summary":"","title":"100boxes.com","type":"tags"},{"content":"I would like to announce the release of a new project I have been working on. 100Boxes.com FREE online Football Squares Pool . I basically took the classic superbowl squares game, that is usually played with paper and spreadsheets and I put it online. I tried to make it as simple as possible for someone to manage their pool, and for people to buy their squares. I started the project pretty late in the season, and unfortunately didn\u0026rsquo;t release it in time for a good test group this year.\nI will continue to work on it over the next few months and hopefully it will be 100% ready for the start of the next football season. I hope to add some cool new features that aren\u0026rsquo;t available anywhere else, and should take the typical game, and bring it to the next level.\nThe game was written in python using the Django framework, and MySQL for a database.\nCheck it out if you have time, and let me know what you think.\n","date":"February 1, 2009","externalUrl":null,"permalink":"/2009/02/01/100boxes.com-free-online-football-squares-pool-management","section":"Posts","summary":"","title":"100Boxes.com - Free Online Football Squares Pool Management","type":"post"},{"content":"Apache2 comes pre installed on Mac OS X Leopard as does Python (2.5). But it doesn\u0026rsquo;t come with mod_python. Here is what you need to do to get it to work.\ndownload the latest file from http://www.apache.org/dist/httpd/modpython/ I downloaded (mod_python-3.3.1.tgz)\n$ tar xvzf mod_python-3.3.1.tgz\n$ cd mod_python-3.3.1\n$ ./configure \u0026ndash;with-apxs=/usr/sbin/apxs\nNow we need to patch the make file so that it will work correctly. edit src/MakeFile\nAdd -arch x86_64 -arch ppc -arch i386 to the end of the LDFLAGS line.\nAdd -arch x86_64 -arch ppc -arch i386 to the end of the CFLAGS line.\nUnder the mod_python.so target add the following to the line after the -c -Wc,\u0026quot;-arch x86_64\u0026quot; -Wc,\u0026quot;-arch ppc\u0026quot; -Wc,\u0026quot;-arch i386\u0026quot; so my complete line looks like. $(APXS) $(INCLUDES) -c -Wc,\u0026quot;-arch x86_64\u0026quot; -Wc,\u0026quot;-arch ppc\u0026quot; -Wc,\u0026quot;-arch i386\u0026quot; $(SRCS) $(LDFLAGS) $(LIBS)\nWhen that is done:\n$ make\n$ sudo make install\nand then:\n$ cd /etc/apache2\n$ sudo cp httpd.conf httpd.conf.orig\n$ sudo emacs /etc/apache2/httpd.conf\nAdd the following to the httpd.conf file.\nLoadModule python_module libexec/apache2/mod_python.so\nRestart apache\n$ sudo /usr/sbin/apachectl restart\nTest configuration\nsudo /usr/sbin/apachectl -t\nThat is all, if there are no errors, it worked!\n","date":"January 23, 2009","externalUrl":null,"permalink":"/blog/2009/01/installing-mod_python-on-mac-os-x-leopard-10.5.x/","section":"Posts","summary":"","title":"Installing mod_python on Mac OS X Leopard (10.5.x)","type":"post"},{"content":"","date":"January 23, 2009","externalUrl":null,"permalink":"/tags/mod_python/","section":"Tags","summary":"","title":"Mod_python","type":"tags"},{"content":"","date":"January 23, 2009","externalUrl":null,"permalink":"/tags/osx/","section":"Tags","summary":"","title":"OSX","type":"tags"},{"content":"I came across a query that seemed like it should be pretty easy, but it wasn\u0026rsquo;t.\nSuppose you have the following Model Class.\n# pseudo code.\nclass Person(models.Model):\nfirst_name = models.CharField()\nlast_name = models.CharField()\nI would like the query to return all Person\u0026rsquo;s who have the same first_name and last_name\nMy first guess was to do the following.\nperson_list = Person.objects.filter(first_name=last_name)\nbut that doesn\u0026rsquo;t work because it wants to have last_name as a variable.\nThis is what I ended up doing.\nperson_list = Person.objects.extra(where=[\u0026lsquo;first_name=last_name\u0026rsquo;]\nYou could also do it using a models.Manager, if that is what you prefer.\nEven though this is an edge case, I\u0026rsquo;m not sure why Django doesn\u0026rsquo;t support this out of the box, but I am pretty sure there is a pretty good reason.\n","date":"January 21, 2009","externalUrl":null,"permalink":"/blog/2009/01/django-query-using-value-from-another-field-in-same-table/","section":"Posts","summary":"","title":"Django Query using value from another field in same table","type":"post"},{"content":"","date":"January 21, 2009","externalUrl":null,"permalink":"/tags/models/","section":"Tags","summary":"","title":"Models","type":"tags"},{"content":"","date":"January 21, 2009","externalUrl":null,"permalink":"/tags/query/","section":"Tags","summary":"","title":"Query","type":"tags"},{"content":"","date":"January 17, 2009","externalUrl":null,"permalink":"/tags/django-settings/","section":"Tags","summary":"","title":"Django Settings","type":"tags"},{"content":"I don\u0026rsquo;t know about you, but when I typically work on a Django application I have at least two environments (Local and Production), and a lot of times I am working on these projects with at least one other person.\nThe way that django currently supports project settings is to have all settings in a settings.py file. This works fine if you are working by yourself in one environment, but once you add something else, it makes managing the settings a pain. One solution, which isn\u0026rsquo;t very flexible is to make all environments have the same settings. That solution doesn\u0026rsquo;t usually fly, so I prefer another approach.\nI take my settings file and only put settings that will be the same for every install in there. Then I create a local_settings.py file in the same directory and put only settings that are environment specific in that file. Each environment is responsible for providing their own file and setting their own settings.\nOne important thing to remember, NEVER put the local_settings.py file in source control or else it will just be a matter of time before someone accidentally checks in their file and breaks everyone else. I typically do a Subversion ignore on the file.\nI also create a file called local_settings.py.template that has all required settings and examples, and stick that one in source control. When you are setting up a new environment all you need to do is copy the local_settings.py.template and change the values for your ENV. Make sure you keep the local_settings.py.template file up to date.\nshort example of settings.py and local_settings.py\n\u0026raquo; settings.py\nimport local_settings\n# there is more above that I removed\nDATABASE_ENGINE = local_settings.DATABASE_ENGINE DATABASE_NAME = local_settings.DATABASE_NAME DATABASE_USER = local_settings.DATABASE_USER DATABASE_PASSWORD = local_settings.DATABASE_PASSWORD DATABASE_HOST = local_settings.DATABASE_HOST DATABASE_PORT = local_settings.DATABASE_PORT # there is more below that I removed\n\u0026raquo; local_settings.py\n# there is more above that I removed\nDATABASE_ENGINE = \u0026lsquo;mysql\u0026rsquo;\nDATABASE_NAME = \u0026lsquo;dbname\u0026rsquo; DATABASE_USER = \u0026lsquo;dbusername\u0026rsquo; DATABASE_PASSWORD = \u0026lsquo;dbpassword\u0026rsquo; DATABASE_HOST = \u0026rsquo;\u0026rsquo; DATABASE_PORT = \u0026lsquo;\u0026rsquo;\n# there is more below that I removed\n","date":"January 17, 2009","externalUrl":null,"permalink":"/blog/2009/01/managing-django-settings-across-multiple-environments/","section":"Posts","summary":"","title":"Managing Django settings across multiple environments","type":"post"},{"content":"I keep forgetting all of the different django commands so I am creating this cheat sheet for myself.\n# start a new project\n\u0026raquo;\u0026gt; django-admin.py startproject $mysite\n# create a new site inside of a project\n\u0026raquo;\u0026gt; python manage.py startapp $siteName\n# start dev server\n\u0026raquo;\u0026gt; python manage.py runserver\n# sync the database with recent model changes.\n\u0026raquo;\u0026gt; python manage.py syncdb\n","date":"May 24, 2008","externalUrl":null,"permalink":"/blog/2008/05/django-common-commands/","section":"Posts","summary":"","title":"django common commands","type":"post"},{"content":"","date":"March 9, 2008","externalUrl":null,"permalink":"/tags/htmlworker/","section":"Tags","summary":"","title":"HTMLWorker","type":"tags"},{"content":"","date":"March 9, 2008","externalUrl":null,"permalink":"/tags/itext/","section":"Tags","summary":"","title":"IText","type":"tags"},{"content":"iText provides a way to convert HTML snippets to PDF using the HTMLWorker class. Unfortunately there isn\u0026rsquo;t much documentation on what you can do for styles. In order to figure this out I went through the source code. I\u0026rsquo;m putting it here so that I won\u0026rsquo;t have to do that again, and so that others will be able to gain something from my pain.\nThis is from iText 2.0.8\nHTMLWorker supported tags \u0026ldquo;ol ul li a pre font span br p div body table td th tr i b u sub sup em strong s strike h1 h2 h3 h4 h5 h6 img\u0026rdquo;\nTagAttribute [iText methods]Values\npalign [setAlignment()]center, right, justify\npleading [setLeading()]float or float, float\npbefore [setSpacingBefore()]float\npafter [setSpacingAfter()]float\npextraparaspace [setExtraParagraphSpace()]float\nIf you find any more let me know and I\u0026rsquo;ll add them to the list.\nKen\n","date":"March 9, 2008","externalUrl":null,"permalink":"/blog/2008/03/itext-htmlworker-style-guide/","section":"Posts","summary":"","title":"iText HTMLWorker Style guide","type":"post"},{"content":"","date":"March 9, 2008","externalUrl":null,"permalink":"/tags/java/","section":"Tags","summary":"","title":"Java","type":"tags"},{"content":"","date":"March 9, 2008","externalUrl":null,"permalink":"/tags/pdf/","section":"Tags","summary":"","title":"PDF","type":"tags"},{"content":"","date":"March 9, 2008","externalUrl":null,"permalink":"/tags/style/","section":"Tags","summary":"","title":"Style","type":"tags"},{"content":"I just released a new version of CampRate.com and here are some of the new features / changes.\nI am about 90% finished with the new image uploader, it uses AJAX to make it look cool.. This just needs to be tested some more, and will be released when we finish with the new add/edit campground flow.95% finished with the PayPal upgrade. Just need to do some more testing. When complete, people can upgrade their listings to add new features using paypal.Added links to the siteMap page to the google sitemap file, and the yahoo sitemap file.finished working on the Latitude/longitude script, given an address it runs and gets the lat and long and updates the db..Made it easier to add a maintainer to a campground when you don\u0026rsquo;t have access to the database.When turned on basic listings will have only 1 image.Setup an efax account so that we can get free incoming fax\u0026rsquo;sIf you want to do something that requires you to login, but you need to register, it will now remember what you were doing before you registered, and when complete will forward you to that page.added link to add Campground reviews RSS Feed to Google Reader.Created a Credits page where we can give credit to tools and people that helped us create CampRate.comChanged it so that it says USA instead of campground list in the breadcrumbsFinished adding the rest of the state imagesAdded a new SlideShow option to view the campground images.\nCampRate.com - Campground Reviews\n","date":"April 23, 2006","externalUrl":null,"permalink":"/blog/2006/04/new-version-of-camprate.com-is-now-available/","section":"Posts","summary":"","title":"New version of CampRate.com is now Available","type":"post"},{"content":"** This Post is a work in progress, and is still being worked on, it hasn\u0026rsquo;t been edited yet, and may contain errors, use at your own risk **\nOver the course of the last year or so I have been hearing a lot about AJAX, and how cool it is, and what neat stuff you can do with it. At first I wasn\u0026rsquo;t too impressed, mostly because I have heard it all before. It wasn\u0026rsquo;t until I saw google maps that I realized how cool it was. I have been itching to use Ajax ever since.\nMy first attempt at AJAX was using the Google Maps API to add a map to the search results page on CampRate.com. It is fairly simple, the search allows the visitor to search for campgrounds given a zip code and a search radius. I search the CampRate.com database and return the campgrounds in the area, and I then add Site Markers onto the google map so that it is easier to see where the campgrounds are located. This was fairly easy to setup given the fact that Google opened up their API.\u0026lt;/span\nOnce I finished the Google Map hack, I was looking for something else to do. After looking around for a little while I thought of the perfect thing to AJAXify. On CampRate.com we allow Campground Owners to upload images of their campgrounds to the site, and campers are also allowed to upload images of their camping trips for everyone to see. The interface for this is very simple, it is just a regular File browse box and a submit button, once they upload the image, the browse box goes away and a thumbnail of the image is inserted. This is a very basic and a very late 90\u0026rsquo;s way of doing things. What a prefect thing to AJAXify.\nOnce I knew what I was going to work on, I needed to figure out how I could improve on the design. After a couple of revisions I came up with the new design and layout. One of the features that I thought would be good to use AJAX for, is a progress bar for showing the status of the upload.\nHere is what the new upload page looks like, this is where I would like to add a progress bar.\nProgress Bar.\nBefore I started coding for the progress bar, I decided to search around to see if anyone else had used a progress bar before. I found a bunch of examples, PHP, Ruby, and a good one using Java + DWR. I decided to look at the Java + DWR example since I was already using Java and DWR on CampRate.com. The Java + DWR example was a great start, the only problem was that it didn\u0026rsquo;t use Struts and once the file was uploaded it changed pages. There were some comments associated with the article that mentioned how to get it to work with Struts, so I decided to take the source code from the article and play with it and see if I could get it to work with Struts.\nWhat I wanted to change in the example.\nGet the example to work with strutsget the file to upload to a selected directoryshow the status of the file in the progress meterwhen the file is uploaded don\u0026rsquo;t leave the page, stay on the same page (no redirect or refresh).What I did.\nI downloaded the war file with source code from the articlerefactored the project directorychanged the build scriptsetup struts, log4jadded logging statements for easier debuggingAll of the new struts code that I added was put into a new package net.kencochrane so that you can easily see what is was originally there, and what was added in order to get struts to work with the original Code.\nStruts Changes\nAdded the struts libs to the lib dir in the projectadded the XML files to the config directorychanged web.xml so that it works with strutscreated some actions, and form\nCode Changes\nAdded a MultipartRequestHandleradded struts actions to handle all requests.The example now actually stores the uploaded files to a temp directory.Added a new package net.kencochrane\nUploading a file without leaving the page\nI looked around to find a way to upload the files without leaving the page, but from what I could find, I guess you can\u0026rsquo;t do this with javascript because of security reasons. The most common way to do this was to use an iFrame and use this iFrame to hold the upload form, and when the form is submitted the form gets posted like normal, and using some tricks with javascript you can make it look like the file is uploaded without ever leaving the page.\nI changed the index.jsp file to include an iFrame, the iFrame holds the upload form. Using javascript I was able to make it looks like the files are being uploaded without leaving the page.\nHere is what the new pages look like, the first image looks like the original form, the only difference is that the page is now using an iFrame to hold the upload form.\nFile upload in progress, it displays the progress bar until it is complete, and then the bar goes away and is replaced with a status message.\nThe file is finished, progress bar is hidden, and the status message tells you the file status.\nWhat is left to do\nadd better error checkingCheck for file being uploaded that is too large.\nNow What\nNow that I have the example working the way I want it (with struts, DWR, and having the file uploaded without leaving the page.) I can work on adding this code to CampRate.com.\nThe more that I think about it, I could also use this on PopcornMonsters.com as well, we allow people to add images for movies, and actors to the site, but the interface is very ugly, once I get the code written for CampRate.com I could easily spend the time and get the same code to work on PopcornMonsters.com, that would make the whole adding images experience so much better.\nSpecial Thanks to Pierre-Alexandre Losson for writing the original Article and for posting his code for everyone to use. I will update this post with some completed pictures once I finish up with all of the code. Feel free to post comments.\nKen Cochrane\nhttp://KenCochrane.net\n","date":"March 21, 2006","externalUrl":null,"permalink":"/2006/03/ajax-struts-file-upload-progress-meter/","section":"Posts","summary":"","title":"AJAX Struts File Upload Progress Meter","type":"post"},{"content":"Debian Package management\napt-get update\nRun this after changing /etc/apt/sources.list or /etc/apt/preferences. (this debianhelp.org page is helpful); run it periodically anyway, to make sure your source list is up-to-date.\napt-get install packagename\ninstalls a new package: rpm -i package.rpm\napt-cache search string\nSearches for string in the list of known packages: like using rpmfind.\ndpkg -l package-name-pattern\nList packages matching pattern: rpm -q pattern or rpm -qa | grep pattern.\naptitude\nCurses viewer of packages installed or available.\napt-cache showpkg pkgs\u0026hellip;\nShow information about packages.\napt-cache dumpavail\nPrints out an available list.\napt-cache show pkgs\u0026hellip;\nDisplays package records, similar to dpkg \u0026ndash;print-avail. Like rpm -q packagename.\napt-cache pkgnames\nFast listing of every package in the system.\ndpkg -S file\nWhich installed package owns the file? rpm -qf file.\ndpkg -L package\nList files in the package. rpm -ql package\napt-file search filename\nSearch for a package (need not be installed) containing files including the string. apt-file is a package of its own, which you may have to apt-get install first, then run apt-file update. If apt-file search filename shows you too much, try apt-file search filename | grep -w filename (which shows you only the files that contain filename as a whole word) or variants like apt-file search filename | grep /bin/ only files located in directories like /bin or /usr/bin, useful if you\u0026rsquo;re looking for a particular executable).\napt-get autoclean\nRun this periodically to clean out .deb archives from packages which are no longer installed on the system. You can regain lots of disk space that way. If you\u0026rsquo;re really desperate for disk space, apt-get clean is more radical, and will remove .deb files even for packages currently installed. But most of the time you probably don\u0026rsquo;t need the .debs any more, so it might be worth it if you\u0026rsquo;re strapped for megabytes.\napt-cache policy pkg\nShow which repository pkg came from, if you have multiple ones in sources.list.\napt-cache show $package | grep ^Source\nFind the source package which produces this binary package. If it returns nothing, then the source package name is the same as the binary package name.\napt-cache depends $package\nShow what $package depends on.\napt-cache rdepends $package\nShow what other packages depend on $package. grep-excuses $package$\nShow why package, in unstable, isn\u0026rsquo;t in testing yet.\ndeborphan and debfoster are great for finding orphaned and unneeded packages which can be removed.\nYou can pull from a different repository by editing /etc/apt/sources.list to replace \u0026ldquo;stable\u0026rdquo; with \u0026ldquo;unstable\u0026rdquo; (or whatever) then doing apt-get update.\n","date":"November 13, 2005","externalUrl":null,"permalink":"/blog/2005/11/debian-package-management/","section":"Posts","summary":"","title":"Debian Package management","type":"post"},{"content":"Last summer a bunch of friends and I decided we would go camping. I have been camping before, when I was a kid, so I knew what I was getting myself into. First off I had to find out if I had all of the proper equipment. I found my Tent, sleeping bag, propane grill and some big blue tarps. Once I gathered all of my stuff and dusted it off, I needed to check with my friends to see if they had everything they needed as well. After a few weeks we all gathered all of the stuff we needed, and all agreeed on a weekend that worked for everyone. Once we had that done, we thought it would be a piece of cake to find a campground to fit our needs. Were we wrong.\nThe first thing we did was hit the internet, we found a bunch of websites for campgrounds in Maine, but they were all marketing hype, no real testimonials from real campers that we could find. We eventually narrowed it down to 3 or 4 different campgrounds and we just had to pick. We picked the one we thought was the best fit for us, in the traveling range that we agreed upon (less than 1 hours drive). We went camping and we used my big blue tarps more then anything else, since it rained off and on the whole weekend. Other than that it was a pretty good experience.\nAs my friends and I were talking around the campfire, we were wondering why in this day and age, it took us so long for us to find a good campground. We realized that there wasn\u0026rsquo;t any one website out there where we could go to, and search all of the campgrounds at once, and compare what they had, and then read reviews from people who have actually been there. Since we were all in the software industry, and we all have developed many websites, we decided this might be a good website to develop. It would help people find a campground that they want faster, and you can actually see if the campground is as good as they say it is, on their website. That is how CampRate.com was born. We wanted to create a website that would list all of the campgrounds in the US and the visitor could pick a state or zip code and search for what amenities that they wanted to have in a campground. Once they got a list back, they could then sort by rating type. For instance, if you wanted good clean bathrooms, then you could sort your results so that all of the campgrounds that had very clean bathrooms come up first. You could also do this for other areas such as Pet and Kid Friendly campgrounds. We figured campgrounds would like it as well since they would be able to see what their visitors are saying, and if they are saying something bad, such as the bathrooms are not clean, then they know that they better clean up their bathrooms or they could lose more customers.\nWe released CampRate.com on June 1st, 2005 and we are still adding features, and making it look nicer, but right now we are only adding campgrounds for the state of Maine. Once we get all of those added we will slowly move out to the different states, growing slowly until we have the whole USA. One day we hope to make it so that picking a campground won\u0026rsquo;t be so hard, and you could then spend more of your valuable time doing more important things. If you get a chance you should check us out, and if you can think of anything that would make our site better, please let us know.\nCampRate.com - Campground Directory and Reviews\n","date":"June 12, 2005","externalUrl":null,"permalink":"/blog/2005/06/camprate.com-online-campground-directory-with-reviews/","section":"Posts","summary":"","title":"CampRate.com  online campground directory with reviews","type":"post"},{"content":"If you accidentally browse up to the root directory or some other level that is too large for WinCVS to initially deal with and you have the flat mode browser view toggle button of WinCVS on, WinCVS will freeze indefinitely. If killing it in Task Manager and restarting doesn\u0026rsquo;t fix your problem, you need to hack the registry.\nLaunch regedit (Start -\u0026gt; Run \u0026ldquo;regedit\u0026rdquo;) and browse to:\nHKEY_CURRENT_USER/Software/WinCvs/wincvs/CVS settings\nIn CVS settings, find the variable P_DisplayRecursive in the right hand window. Right-click on P_DisplayRecursive and select Modify from the right-click menu. in the window that gets displayed, change the Value data set with only two numbers in it from the 01 to 00. Then click Ok at the bottom of the window, shut down Registry Editor, start up WinCVS again, and you should now have the flat mode button toggled off.\nSome companies don\u0026rsquo;t let their employees run regedit, but do allow employees to run .reg files. If so, put the following into a file called Unflatten.reg, run it, it should work for you:\nWindows Registry Editor Version 5.00 /S\n[HKEY_CURRENT_USER\\Software\\WinCvs\\wincvs\\CVS settings]\n\u0026ldquo;P_DisplayRecursive\u0026rdquo;=hex:00 ","date":"May 5, 2005","externalUrl":null,"permalink":"/blog/2005/05/getting-wincvs-out-of-flat-mode/","section":"Posts","summary":"","title":"Getting WinCVS out of flat mode","type":"post"},{"content":"\nImporting (Restoring) Many MySQLDump Text Files:\n~ from a user (paraphrased): \u0026ldquo;If we have 100+ mysqldump .txt files, how can we easily import them all, in order to restore a database, without typing each file name in by hand? (i.e. from the shell prompt, piped to the mysql monitor:\nmysql mydb \u0026lt;\u0026gt;). Wouldn\u0026rsquo;t it be better for the script to write all the tables to one text sql file?\u0026rdquo; =\u0026gt; I suppose the script could be rewritten, but it\u0026rsquo;s more flexible the way it is, considering that one can use the Linux shell prompt to do the same thing. Assuming that the database name is \u0026rsquo;test\u0026rsquo;, and that the files have the extension of \u0026lsquo;.txt\u0026rsquo;, you can use this command:\nfind . -name \u0026ldquo;.txt\u0026rdquo; -print | xargs -t \u0026ndash;replace cat {} | mysql test\nThat parses through all the files and reads the contents using \u0026lsquo;cat\u0026rsquo; and then pipes the contents one by one to the mysql monitor with the database \u0026rsquo;test\u0026rsquo;. The \u0026lsquo;-t\u0026rsquo; shows you the file names as it works, so you\u0026rsquo;re not staring at a blank screen. If you change the \u0026lsquo;-t\u0026rsquo; to a \u0026lsquo;-p\u0026rsquo;, it will prompt you at each file. Try it on a test database first, to be safe :-) Note: I think the xargs version above is better than:\ncat .txt | mysql test\n\u0026hellip; because of the limitations of the command line size, assuming one has many files. Also, the xargs version shows you the filenames as they process. Of course, there are probably many other ways to accomplish the above, given the eclectic nature of Unix.\nfind . -name \u0026ldquo;.sql\u0026rdquo; -print | xargs -t \u0026ndash;replace cat {} | /usr/local/mysql/bin/mysql \u0026ndash;password -D pm\nhttp://worldcommunity.com/opensource/utilities/mysql_backup.html\nSecure copy file from one host to RemoteHost:\nscp $FileName $UserName@$RemoteHost:/path/on/remote/host/$FileName\n","date":"February 13, 2005","externalUrl":null,"permalink":"/blog/2005/02/commands-i-use-but-often-forget/","section":"Posts","summary":"","title":"Commands I use, but often forget","type":"post"},{"content":"If you need to connect to a MySQL 4.1 Database using an old mysql library, you need to change the password for the user account that you want to connect with, so that it uses an old password. The new version of the database uses a different password algorithm which is much longer then the old version, which causes problems with the old library. To fix this you have to set the user password with the Old_password( ) function.\nSET PASSWORD FOR \u0026lsquo;some_user\u0026rsquo;@\u0026rsquo;some_host\u0026rsquo; = OLD_PASSWORD(\u0026lsquo;mypass\u0026rsquo;); http://dev.mysql.com/doc/mysql/en/password-hashing.html\n","date":"February 13, 2005","externalUrl":null,"permalink":"/blog/2005/02/connecting-to-mysql-4.1-with-old-libraries/","section":"Posts","summary":"","title":"Connecting to MySQL 4.1 with Old Libraries","type":"post"},{"content":"This is an install guide for setting up a secure mail server using Postfix, Courier-IMAP, MySQL, and Postfixadmin on Debian Woody. Before we begin here are a few notes regarding how things are handled: Virtual domains - All domains including the local domain are handled as virtual domains. This allows a central location for all mail. All interactions with the server are authenticated and encrypted using libsasl2 and TLS/SSL (Please see the packages section for what to install) 1. Packages to Install This install guide assumes the following packages are already installed: mysql-clientmysql-serveropenssl (to create the certificate)apache To add Postfix add the following line to your /etc/apt/sources.list deb http://www.backports.org/test/postfix/ ./ Install the following packages: Postfix 2.1.4-4 Postfix-mysql 2.1.4-4 Postfix-pcre 2.1.4-4 Postfix-tls 2.1.4-4libsasl2 (the Cyrus SASL library) libsasl2-modules libsasl2-modules-sql Note: Make sure you use the source above to install the packages. DON\u0026rsquo;T use the stable version of backports to install Postfix and libsasl. The stable version of Postfix is linked against libsasl1 which does not allow authentication via the libsasl2 sql module. Comment out the previous package line and add the following line in your sources.list file: deb http://www.backports.org/debian stable courier Install the following packages: courier-authdaemon 0.47-2 courier-authmysql 0.47.2 courier-base 0.47.2 courier-imap 3.0.8-2 courier-imap-ssl 3.0.8-2 Finally unpack the latest version of Postfixadmin (2.1.0 as of this writing) into your web directory. 2. Create the Tables in MySQL Go to the postfixadmin directory that you unpacked and run the DATABASE_MYSQL.TXT in MySQL to setup the complete table structure for Postfix. It should create the following 7 tables under a database called \u0026lsquo;postfix\u0026rsquo;: admin alias domain domain_admins log mailbox vacation 3. Create a Mail Directory Create a directory to have all your virtual users mail dropped in, this directory needs to be owned by Postfix. % mkdir /usr/local/virtual % chown -R postfix:postfix /usr/local/virtual % chmod -R 771 /usr/local/virtual 4. Postfix Configuration Use the following /etc/postfix/main.cf file. The items in \u0026lt;\u0026gt; are values you need to change. myhostname= \u0026laquo;span style=\u0026ldquo;font-style: italic;\u0026quot;\u0026gt;your.hostname\u0026gt; mydomain= \u0026laquo;span style=\u0026ldquo;font-style: italic;\u0026quot;\u0026gt;your.domain\u0026gt; mydestination = myorigin = /etc/mailname virtual_maps = mysql:/etc/postfix/mysql_virtual_alias_maps.cf virtual_alias_maps = $virtual_maps virtual_gid_maps = static:\u0026laquo;span style=\u0026ldquo;font-style: italic;\u0026quot;\u0026gt;postfix group id from /etc/passwd\u0026gt; virtual_mailbox_base = \u0026laquo;span style=\u0026ldquo;font-style: italic;\u0026quot;\u0026gt;/your/mailbox/path\u0026gt; virtual_mailbox_domains = mysql:/etc/postfix/mysql_virtual_domains_maps.cf virtual_mailbox_maps = mysql:/etc/postfix/mysql_virtual_mailbox_maps.cf virtual_minimum_uid = \u0026laquo;span style=\u0026ldquo;font-style: italic;\u0026quot;\u0026gt;postfix user id from /etc/passwd\u0026gt; virtual_transport = virtual virtual_uid_maps = static:\u0026laquo;span style=\u0026ldquo;font-style: italic;\u0026quot;\u0026gt;postfix user id from /etc/passwd\u0026gt; # SMTP Authentication smtpd_sasl_auth_enable = yes broken_sasl_auth_clients = yes smtpd_sasl_security_options = noanonymous smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reje ct_unauth_destination smtpd_use_tls = yes smtpd_tls_cert_file = /etc/postfix/smtpd.cert smtpd_tls_key_file = /etc/postfix/smtpd.key append_dot_mydomain = no Notes: Leave mydestination blank if you want your local domain to be considered virtual. If you list your local domain in mydestination, as is commonly the case, you need to have a system user for every email user. Postfix will look for a system user corresponding to the email user and if none is found it will bounce the mail. It will also not deliver to the mailbox path. Create the following files in /etc/postfix: mysql_virtual_alias_maps.cf user = postfix password = hosts = localhost dbname = postfix table = alias select_field = goto where_field = address mysql_virtual_domains_maps.cf user = postfix password = hosts = localhost dbname = postfix table = domain select_field = description where_field = domain mysql_virtual_mailbox_maps.cf user = postfix password = hosts = localhost dbname = postfix table = mailbox select_field = maildir where_field = username Now go to /etc/postfix/master.cf and change first smtp line to the following: smtp inet n - n - - smtpd This prevents postfix from running in a chrooted jail. 5. Postfix, Authenticated SMTP and TLS encryption By default postfix only allows communication between users defined in mynetworks. This prevents an open relay that spammers can abuse to send out spam. This however, also prevents users of your domain to send out email to domains outside your box. So you have to authenticate your users with a username and password to allow them to be part of your circle of trust. Most email cilents such as Thunderbird have this feature included. Use the following from Christoph Haas\u0026rsquo;s tutorial located in http://workaround.org/articles/ispmail-sarge/ to setup authentication: As written earlier Postfix uses the Cyrus SASL library for authenticated SMTP. So you need to tell Postfix how to access the data storage that keeps the usernames and passwords. This is easy. Create a directory /etc/postfix/sasl to put the SASL config file there: mkdir /etc/postfix/sasl Now you need to create a file smtpd.conf in that directory like this: pwcheck_method: auxprop auxprop_plugin: sql allowanonymouslogin: no mech_list: plain login cram-md5 digest-md5 sql_engine: mysql sql_hostnames: localhost sql_user: postfix sql_passwd: sql_database: postfix sql_select: select password from mailbox where username=\u0026rsquo;%u@%r\u0026rsquo; (If you have trouble with SASL you may consider inserting a line like \u0026ldquo;log_level: 7\u0026rdquo; here. It will write more verbose information to the log files and perhaps help to find the cause.) An important step is to encrypt the SMTP session. Otherwise the username and password could be transmitted in a very insecure way if the mail client chose to use one of plaintext authentication methods). So I encourage you to encrypt that communication using TLS. TLS is short for Transport Layer Security (RFC2246) and in short terms uses SSL (Secure Socket Layer) which encrypts the mail connection between the road-warrior and the mail server. First you will need an SSL certificate. If you don\u0026rsquo;t want to pay for one from your favorite trustcenter you can well use a self-signed one. (Personal note: I wonder how paying for something makes it more trusted.) The only drawback: the mail client does not know about your CA (certificate authority) and will spit out a warning to the user. Either tell the users to ignore the warning or let them install the certificate on their computers. For a certificate that is valid for one year for the hostname smtp.domain.tld you would type this: openssl req -new -outform PEM -out smtpd.cert -newkey rsa:2048 -nodes -keyout smtpd.key -keyform PEM -days 365 -x509 You will then be asked a few question about the fields of the certificate. It does not matter what you enter. Just fill the fields. One exception though - the \u0026ldquo;Common Name\u0026rdquo; must be the hostname of your mail server. Example session: Country Name (2 letter code) [AU]:DE State or Province Name (full name) [Some-State]:Hamburg Locality Name (eg, city) []:Hamburg Organization Name (eg, company) [Internet Widgits Pty Ltd]:workaround.org email services Organizational Unit Name (eg, section) []:Master of Disaster Common Name (eg, YOUR name) []:smtp.domain.tld Email Address []:postmaster@domain.tld After a short moment you will get two files: \u0026ldquo;smtpd.key\u0026rdquo; (the private key file) and \u0026ldquo;smtpd.cert\u0026rdquo; (the certificate). Move these two files into /etc/postfix. Make sure at least the key file is not readable for the whole wide world: chmod o= /etc/postfix/smtpd.key 6. Configuring Courier-IMAP-SSL Edit /etc/courier/authdaemonrc and change authmodulelist to \u0026ldquo;authmysql\u0026rdquo; like this: authmodulelist=\u0026ldquo;authmysql\u0026rdquo; Add or Change /etc/courier/authmysqlrc to contain the following: MYSQL_SERVER localhost MYSQL_USERNAME postfix MYSQL_PASSWORD MYSQL_SOCKET MYSQL_PORT 3306 MYSQL_DATABASE postfix MYSQL_USER_TABLE mailbox #MYSQL_CRYPT_PWFIELD password #Comment this out MYSQL_CLEAR_PWFIELD password MYSQL_UID_FIELD \u0026lsquo;\u0026rsquo; MYSQL_GID_FIELD \u0026lsquo;\u0026rsquo; MYSQL_LOGIN_FIELD username MYSQL_HOME_FIELD \u0026lsquo;/location/to/mail/dir\u0026rsquo; MYSQL_NAME_FIELD name MYSQL_MAILDIR_FIELD maildir Notes: Make sure you have no spaces only tabs. Enclose user id and gid in single quotes. 7. Final Comments By default libsasl2 only supports clear text passwords so make sure the user passwords in the mysql database are readable. Since our connections are encrypted using TLS/SSL this is not much of an issue. Make sure Postfixadmin is set for clear text passwords. Your config.inc.php in your Postfixadmin directory should have the following: $CONF[\u0026rsquo;encrypt\u0026rsquo;] = \u0026lsquo;cleartext\u0026rsquo;; 8. Helpful Links and Howto\u0026rsquo;s Much of the information for this guide was gathered from the following links: http://high5.net/howto/ http://workaround.org/articles/ispmail-sarge/ http://brunny.com/content/view/12/50/ ","date":"February 8, 2005","externalUrl":null,"permalink":"/blog/2005/02/setting-up-postfix-tls-courier-imap-ssl-mysql-postfixadmin/","section":"Posts","summary":"","title":"Setting up Postfix (TLS) / Courier-IMAP (SSL) / MySQL / Postfixadmin","type":"post"},{"content":"When it comes to Linux, nothing comes easy for me, but I get it done. Tonight I tried to install the connector between Apache and Tomcat. I had basic instructions for that, and he even e-mailed me the required mod_jk.so file. Unfortunately, Mandrake 10 didn\u0026rsquo;t like it and I had to manually build it from the source. What a pain, but the instruction below helped quite a bit, I just had to install \u0026ldquo;autoconf\u0026rdquo; to get it to work. Mandrake asked for the installation CD to be inserted. I already had the other components (gcc).\nThen another late night saga began. It\u0026rsquo;s all in the details. The generic instructions didn\u0026rsquo;t work for me, as usual. So, below you\u0026rsquo;ll see my comments and the little I had to modify to get this to work. FYI, it took me again 4 hours to get this thing going. There are dozens of different configurations available out there, the trick is to find one that works. Finding the problem was also hard, but now it looks so trivial when one knows what is wrong.\nMoral of the story: no pain, no gain.\nIntegrating Apache 2.0.x and Tomcat 5.0.x with mod_jk on Linux (Using FlashGuide as a base.)\nThis FlashGuideTM covers integrating Apache 2.0.x and Tomcat 5.0.x on Unix via mod_jk. These instructions have been tested on SuSE 9.0, Red Hat 6.2 and Fedora Core 2. There are two ways to integrate Apache and Tomcat: mod_jk and mod_jk2. Mod_jk is the older but more stable version, which supports load balancing and non-standard web application locations. Mod_jk2 is newer, has bugs, and, as of 11/15/04, is no longer being actively developed.\nThese instructions are for the current version of Tomcat 5.0.28, but have worked the same for all previous versions of Tomcat.\nBuilding Apache 2.0.x on Linux Unless you can find a binary distribution of Apache with DSO support enabled, you will have to follow these instructions to build it yourself.\nCheck your prerequisites:\nYou will need GCC installed You will need /usr/ccs/bin and the gcc executables in your $PATH Download the latest Apache source from http://httpd.apache.org/download.cgi - currently, the latest is 2.0.52.\nUnpack the distribution into a development directory (I used /usr/local)\nThe distribution directory will be something like apache_2.0.52\nCd into the distribution directory (e.g. /usr/local/apache_2.0.52)\nConfigure the makefile:\n./configure \u0026ndash;with-layout=Apache \u0026ndash;prefix=/usr/local/apache2 \u0026ndash;enable-module=most \u0026ndash;enable-mods-shared=most\nIf you have to run this configuration again, note that configure saves the latest configure command in config.status Build Apache: make\nInstall Apache: make install\nNote the location of your Apache installation - we will refer to this as $APACHE2_HOME\nInstalling Tomcat 5 on Linux\nDownload the latest Tomcat binary from the Tomcat 5 section of http://jakarta.apache.org/site/binindex.cgi. Currently, Tomcat 5.0.28 is the latest.\nInstall Tomcat by unzipping/untaring the download file and placing in the desired directory (I used /usr/local)\ncd /usr/local tar zxf ./jakarta-tomcat-5.0.28.tar.gz\nNote the location of your Tomcat installation - we will refer to this as $CATALINA_HOME Optionally, save time on typing by creating a symbolic link like this: ln -s jakarta-tomcat-5.0.28 tomcat5 //RK I used just “tomcat”\nInstalling mod_jk on Linux Follow these steps to install a binary copy of mod_jk.\nGo to the JK 1.2 section of http://jakarta.apache.org/site/binindex.cgi and look in the folder for your OS. If you find one, download it. If you don\u0026rsquo;t, skip to the next section.\nRename the downloaded file to mod_jk.so\nCopy mod_jk.so to $APACHE_HOME/libexec\nBuilding mod_jk on Linux\nFollow these steps if you can\u0026rsquo;t find a mod_jk binary or just want to build your own.\nCheck your prerequisites:\nType \u0026lsquo;which libtool\u0026rsquo; to see if libtool in your $PATH. If it is not, and it is really not on your system, you will need the latest libtools from GNU: * Download the latest libtools from ftp://ftp.gnu.org/gnu/libtool/. * Unpack the distribution into the desired directory * Cd into the distribution directory and type the following:\n./configure make make install\nMandrake 10: use software update/install from the KDE to get this done. Will need CD’s 1 \u0026amp; 4 Now type \u0026lsquo;which libtool\u0026rsquo; to make sure that libtool is in your $PATH. If its not, make sure you find out where \u0026lsquo;make install\u0026rsquo; put libtool and libtoolize and add that location to your $PATH. Type \u0026lsquo;which autoconf\u0026rsquo; to see if autoconf in your $PATH. If it is not, and it is really not on your system, you will need the latest autoconf from GNU:\nOn my system, I had to de-install older rpms for autoconf first. * Download the latest autoconf from http://ftp.gnu.org/gnu/autoconf/. * Unpack the distribution into the desired directory * Cd into the distribution directory and type the following: ./configure make make install\nNow type \u0026lsquo;which autoconf\u0026rsquo; to make sure that autoconf is in your $PATH. If its not, make sure you find out where \u0026lsquo;make install\u0026rsquo; put autoconf and add that location to your $PATH. RK: I don\u0026rsquo;t believe Ant is necessary. I already had Ant 1.6.2 installed.\n3. Make sure you have Ant 1.5.1 or greater installed and $ANT_HOME set. If not, follow these steps: * Download the latest Ant (currently 1.6.2) from http://ant.apache.org/bindownload.cgi * Unpack the distribution in the desired directory (I used /usr/local) * Optionally, save time on typing by creating a symbolic link like this: ln -s apache-ant-1.6.2 ant\nSet $ANT_HOME in your environment (or in /etc/profile so you don\u0026rsquo;t forget it next time) to point to the ant installation * Make sure $ANT_HOME/bin is in your $PATH Download the latest JK 1.2 Tomcat Web Server Connectors from http://jakarta.apache.org/site/sourceindex.cgi. As of this writing, JK 1.2.6 is the latest. RK: correction: JK 1.2.8 is now the latest version and it was available for download on 2/7/2005. It wasn\u0026rsquo;t on the prior day (I even sent an e-mail to the Jakarta webmaster). As of December, the download link for JK 1.2 doesn\u0026rsquo;t seem to be working. In the meantime, therefore, you can find JK 1.2.6 on my site. 3. Unpack the distribution in the desired directory\nCurrently, the distribution directory is jakarta-tomcat-connectors-jk-1.2.6-src Cd into the distribution directory Configure the makefile: cd jk/native ./configure --with-apxs=/usr/local/apache2/bin/apxs --enable-EAPI Replace the values of --with-apxs with the appropriate path to the apxs executable in your Apache installation directory. Now do the build: make Finally, install cp ./apache-2.0/mod_jk.so $APACHE2_HOME/modules $APACHE2_HOME refers to the directory in which you installed the compiled Apache 2.x - in my case, it is /usr/local/apache2.\nConfiguring mod_jk for Apache 2.0.x\nCreate workers.properties in $APACHE_HOME/conf with the following contents:\nworkers.tomcat_home=/usr/local/tomcat workers.java_home=$JAVA_HOME ps=/ worker.list=default //rk: this is the worker name to be used in Apache’s httpd.conf\nworker.default.port=8009 worker.default.host=localhost worker.default.type=ajp13 worker.default.lbfactor=1\nNaturally, you will take care to set workers.tomcat_home and workers.java_home as approriate for your environment.\nOften, people put the workers.properties file in $CATALINA_HOME/conf/jk, as my previous FlashGuides have also recommended. However, putting it in $APACHE_HOME/conf is more appropriate since its purpose is to tell Apache about one or more local or remote Tomcat installations. 2. For Apache 1.3.x, edit $APACHE_HOME/conf/httpd.conf and add the following to the LoadModules section:\nLoadModule jk_module libexec/mod_jk.so For Apache 2.0.x, edit $APACHE_HOME/conf/httpd.conf and add the following to the LoadModules section: LoadModule jk_module modules/mod_jk.so Edit $APACHE_HOME/conf/httpd.conf and add the following to the AddModules section (if you have it - Apache 2 and later versions of Apache 1.3.x don\u0026rsquo;t):\nAddModule mod_jk.c RK: don’t need for Apache2 Edit $APACHE_HOME/conf/httpd.conf and add the following before Section 3 in httpd.conf:\n# Mod_jk settings # # JkWorkersFile \u0026quot;conf/workers.properties\u0026quot; JkLogFile \u0026quot;logs/mod_jk.log\u0026quot; JkLogLevel error JkMount /jsp-examples default //RK: default is the worker name JkMount /jsp-examples/* default\nEnd of mod_jk settings # The location of the Jk workers.properties and log files are relative to $APACHE_HOME. If you prefer other locations, make sure you fully qualify the paths.\nIn this test, we are directing all URIs that begin with \u0026ldquo;/jsp-examples\u0026rdquo; to Tomcat. The name \u0026ldquo;default\u0026rdquo; corresponds to the JK worker that we defined in $APACHE_HOME/conf/workers.properties. We define two JK Mounts in order to match the URI of \u0026ldquo;/jsp-examples\u0026rdquo; as well as any that start with \u0026ldquo;/jsp-examples\u0026rdquo;. In a real world situation, we might only direct JSPs or servlets to the JK worker. 6. Find the line in $APACHE_HOME/httpd.conf that starts with \u0026ldquo;Port\u0026rdquo; - make sure you have it pointing to a port that is not in use. Lately, Apache has been coming with the \u0026ldquo;Port\u0026rdquo; directive set to \u0026ldquo;8080\u0026rdquo;, which is the same port that Tomcat will listen on. If you don\u0026rsquo;t have any other web server running, you can set it to \u0026ldquo;80\u0026rdquo; instead. 7. Test httpd.conf by typing the following:\ncd $APACHE_HOME/bin apachectl configtest\nnew item inserted by RK: 5a server.xml\nThe connector will work only if it’s entered like this. Tomcat 5 has a default that doesn’t seem to work. Comment it out and paste the line below. Debuh=”1” should be “0” for production mode.\n\u0026lt;Connector port=\u0026ldquo;8009\u0026rdquo; minProcessors=\u0026ldquo;5\u0026rdquo; maxProcessors=\u0026ldquo;75\u0026rdquo; acceptCount=\u0026ldquo;10\u0026rdquo; debug=\u0026ldquo;1\u0026rdquo; protocol=\u0026ldquo;AJP/1.3\u0026rdquo;/\u0026gt;\nend of inserted item by RK\nTesting\nMake sure no other server is running on the default Tomcat ports of 8005, 8009 and 8080. Make sure no other server is running on the Apache port, which is normally 8080 or 80.\nStart Tomcat first:\ncd $CATALINA_HOME/bin ./catalina.sh start\nAlways start Tomcat first and then start Apache. If you have to bounce Tomcat, remember to take down Apache first and restart it after Tomcat restarts. mod_webapp is very sensitive that way. In fact, some users have reported that it is necessary to put in 10 second delay between starting Tomcat and Apache. 3. Start Apache:\ncd $APACHE2_HOME/bin ./apachectl start\nPoint your browser to http://localhost and verify that you get the default Apache page\nSubstitute \u0026ldquo;localhost\u0026rdquo; for the actual machine name/ip if necessary.\nPoint your browser to http://localhost:8080 and verify that you get the default Tomcat page\nPoint your browser to http://localhost/jsp-examples/ and verify that you get the index page for the Tomcat examples. This will be served by Apache and will indicate that you have completed your integration of Apache and Tomcat successfully - congratulations!\n","date":"February 8, 2005","externalUrl":null,"permalink":"/blog/2005/02/integrating-apache-2.0.x-and-tomcat-5.0.x-with-mod_jk-on-mandrake-linux-10/","section":"Posts","summary":"","title":"Integrating Apache 2.0.x and Tomcat 5.0.x with mod_jk on Mandrake Linux 10","type":"post"}]