View in #pipelines on Slack
@ozzmeister00: Is this the right place to find folks who have strong opinions about build processes and continuous integration?
@bob.w: This is as good a place as any. I have strong opinions that they should exist.
As to which ones to use, and how exactly to use them, I have less strong opinions. Except that they should be used.
@ozzmeister00: we’re rocking Team City and it’s set up to trigger off Perforce commits, but the build script we’re using was written for Jenkins that triggered off a timer
so with our new setup I can know which modules got changed and only build those, as opposed to the old system that would try to Build Everything and fail out on stuff that was already built
but my question then is, what do I do with modules that fail to build?
@bob.w: Track all commits between the last successful build, and email blast everyone on that list with their failure.
@dhruvagovil: yes and make sure to include the logs of the failure with as much information as possible
@bob.w: If you’re doing this from a pre-commit hook, also don’t let the submit go through.
If its a post-submit, keep emailing everyone all the info until the shame is to much and they fix it.
@ozzmeister00: not at pre-commit hooks, sadly
@dhruvagovil: when you guys use Perforce for code, do you have some workflow analagous to Git Flow? where you commit to a branch first, do tests there and then commit to the main branch
@bob.w: Possibly widening the circle of shame if X amount of time passes and builds aren’t fixed.
@dhruvagovil: I know perforce doesn’t really do branches the same way as git, but it’s possible to model that workflow in
@bob.w: You can mimic git-flow pretty well using p4-streams. But there are issues when dealing with binary assets and code / asset dependencies.
@ozzmeister00: let’s assume we’re not doing that
we’ll handle unittests and some simple stuff on the build machine, but otherwise we’re doing pre-commit user testing during code review
@bob.w: So my preferred method is to have an active development branch that is getting continuously worked on by the team, this has CI running builds as commits happen.
Then at the end of every sprint, that shit gets merged into a release branch, any end of sprint bugs get fixed in the release branch and merged back to the development branch.
release is also getting builds from CI, but at a much lower interval because the whole team isn’t banging on it constantly.
When we were doing this at my last job, we’d have other shorter lived feature branches that we’d split off from dev, but those wouldn’t get CI checked unless they were unusually long lived.
Basically if it was living longer than 2 sprints, it probably needed to be checked, otherwise it was up to the feature team to test their shit, and CI would pick it up once it got merged back into dev.
the system worked pretty well, and once everyone got past the growing pains we went from having multi-day checkin bans at the end of our 3-week sprints, to sub-hour.
@ozzmeister00: yeah, we’re definitely not at that scale yet.
@dhruvagovil: yeah that sounds like a good workflow
I think it’s not a matter of scale even, it helped us to work like that even for a single person or two person project
@ozzmeister00: are ya’ll talking about your tech art tools specifically, or your game build system broadly?
@bob.w: that was the game build, but similar system for the tech art tools. Just not as sprint gated
as releases were a bit more arbitrary, and we largely operated tangentially to the sprints.
@dhruvagovil: anything really. the workflows are the same imho for code/assets/builds
@bob.w: But yeah, its really about the workflows and having a clear separation between development code and production / release code.
@dhruvagovil: for example at Sony, even for asset publishes we’d do a QC step and only mark stuff as approved when they passed that QC
@bob.w: Basically the idea is to find the earliest point you can test, with the minimum amount of disruption. So that errors get caught quickly, but your output isn’t being overly constrained waiting on results.
@ozzmeister00: For context, our current process is:
1) Submit CL for Code Review, include testing steps and any unittesting
2) Other TAs review code, and test functionality per testing steps
3) Once approved, TA submits CL to the only branch
4) Build machine triggers and tries to build everything in that CL
@dhruvagovil: you don’t build before submitting to the branch?
what happens if the build fails?
@ozzmeister00: it’s all Python
“build” just means copy code into a folder for us
@bob.w: I would have the build machine working at each of those steps. There is no reason it can’t unshelve a for review change and run the unit tests against it. In fact each update to the review should get an automated pass.
if there are tests that require user interaction, say click these 5 buttons in order, while poking 4 faces and a frog, then yeah that’s a good spot for other TAs to jump in and help as part of the review. But automate as much of this as you can.
@ozzmeister00: I’ve been slowly but surely writing unit tests for our stuff where possible but a lot of the code we’re working on right now is the user-interaction kind
I’ll see what I can do to trigger BM to unit test on review. That sounds fun.
@bob.w: Which is fine, if it doesn’t happen to interact with the unit tests, at least you’ll get a thumbs up from the build machine that nothing got worse in the meantime
@dhruvagovil: yeah ideally, before submitting for review, you can run allow TAs to run a build.
Once submitted, it should always run a build and post the info in the review area.
After it’s approved and merged, it should do another build
@bob.w: p4 has shelve triggers, so if you’re using something like swarm you can just look for the review number and have it tied to that.
@dhruvagovil: that’s mostly because stuff will change at all 3 of those stages, especially if there’s any time passing between them
@bob.w: And if you have a more stable release branch, run a build when anything gets merged there. Basically you want to run tests as the code goes from less to more stable.
@theodox: ahem this is a great thread to be discourse-crossposted… jus’ sayin…
@ozzmeister00: but back to my original question, in the event of a build failure I should be trying to build all possible changes not just stuff that triggered the CI
@adam.pletcher: Hard to say without knowing more about your “stuff” and how isolated each thing is… But offhand it seems like yes?
I know we have a lot of cross-tool dependencies that aren’t always that obvious, so building and testing everything is important. Even our game and tools builds are (generally) strapped together.
@bob.w: You want to build the minimum amount of stuff that your change could potentially impact, and in the face of unknowns, build more. Nothing is quite so bad as a breakage that slipped past CI because that means its in a totally non-obvious spot.
@ozzmeister00: we’ve got some pretty good dependency-tracking, so it’d be easy enough to test upstream and downstream packages
like, if A depends on C, and B depends on A, I’d test A, B and C
alright, I think I’ve got a clearer idea of how to move forward. Thanks y’all!
