Development workflow at FinWorks

This is how I picture our development workflow. Drawing a picture like this helps me to see how we work because it gets a bit too much to hold in my head. I hope this helps to improve our process. Perhaps we can spot big flaws in our thinking. Perhaps we can see where it works well. I hope others understand this and can provide some good input. And I hope it helps someone else to set up a good development process. There probably are quite a few missing links. If someone has comment on what’s missing or requires another picture, I’d be grateful.

I think it is good! It has come a long way and evolved to something that we can be proud of.

FinWorks dev workflow

Of course, there is sooo much here. The details of it overwhelms me mostly.

  1. A cron job on Jenkins pulls changes from the main repo every 10 minutes. If the main branch changed, it kicks off a build process that consists of quite a few Jenkins jobs. We all develop in a branch called development. That could just as well have been master. Legacy.
  2. One of the jobs is to build a Pharo image, using the installed image from http://gforge.inria.fr/frs/download.php/31259/Pharo-1.4-14557.zip. Actually, this is a slight lie. An overnight job does this. The one that runs every 10 minutes copies the overnight load and updates it. We use Metacello to do the load. The same script for the overnight job and the update.
  3. We start with GemStone64Bit2.4.4.4-x86_64.Linux/bin/extent0.seaside.dbf, which contains a shipped version of the open source stuff. This is so we don’t have to bootstrap it with Monticello / Metacello (we used to!). We use the same Metacello command used in Pharo to load our stuff then into GemStone (we call this deploy in our scripts). We then do a backup of that database and store it on the job.
  4. This is actually several Jenkins jobs (>20) because we load balance running tests over some Jenkins slaves. It consists of some domain level unit tests, some SeasideTesting tests and some Selenium RC tests (yes, want to get to Beach Parasol). So, we expect of developers to monitor the Jenkins jobs and fix as problems arise. Normally, I’ll watch Jenkins more carefully if I pushed something. Running all the tests takes a long time (about an hour), with 4 concurrent Jenkins slaves. This should be better!
  5. This job restores the overnight production backup taken and deploys the most recent code on it. This is to check that the deployment went through. It does not really prove that the production site(s) works after a deployment, just that the deployment with some post deployment checks worked.
  6. When a developer starts work, he/she can copy the most recently build image from Jenkins. It usually is quite up to date. It is not necessary to get a new image often, but with my recent attempts to improve the loading process and to upgrade the infrastructure (read Metacello, Seaside, Magritte), it is safer to copy an image daily. Takes about 3 minutes from the office.
  7. Upon getting a new image, I usually get my local working copy up to date. I tend to do this often because there are quite a few changes in the repository during the day. This also merges other people’s changes in. This makes it imperative to do step 8 after this one.
  8. Updating the Pharo image is again the same Metacello command that is used to build the image on Jenkins and GemStone. At this point, we make our changes in Pharo. We use chrome a lot to browse / use / test the application and we run tests in Pharo. There are a lot of tests and we run them selectively in Pharo. We store the loaded git commit id in the image to make sure that what’s loaded in Pharo is consistent with the local working copy.
  9. On a dev machine, we can do the same build as in 3. or 5. We do not use GemTools to edit code and save changes from here. (We typically would just edit here to check something out / set a conditional breakpoint.) I would normally verify / debug something in GemTools and then write tests & run them in Pharo.
  10. Once I’m happy with my changes, I’ll save from Pharo. We’ve got a few World menu items that uses Metacello to find the appropriate packages and save them using a script. We seldom save individual packages using the Monticello tools. We just do a “save all”.
  11. We write all our scripts in Ruby and sometimes changes have to align between Smalltalk and the scripts. We also have images, PDF docs and other resources that we need to edit and save in the project. This is all saved in one git repository.
  12. Pushing to github is usually preceded with step 7. to merge other peoples changes in. This involves solving merge conflicts and sometimes repeating steps 8, 9, 10 and 11 before we can push up to github. This triggers step 1 and the whole process repeats.
  13. We deploy to production in the same way that we load into Pharo and in an empty database, essentially the same as 2, 3, 8 and 9 above. The difference between deploying over a restored production database and extent0 is that it will do migrations after loading the code.

Post a Comment

You must be logged in to post a comment.