Oceans of Wisdom

this does not matter so much, if the bots have app and AWS CLI access to the staging environment/vpc/etc they grind it out in the logs and figure it out

and anything we can do to make the deployments faster is important

yes for this to work staging need to stay up during deployments

the bots have spent all weekend fixing at least the obvious things they broke on staging

at least before I handed over my work to the professionals

is working on a common code base

if they are at least merged, then anyone starting work on something new on Monday

so we need to solve each step of the process at a time

it is going to be Conflict Central

we will collapse into chaos, we won't know what branch to work on

and now we are waiting for you too look at each one before even merging

there are 200 PRs unmerged

we can't have a situation where you come in on Monday

this is weeks away not months away here

from trivial (sure, some), but also normal and some serious ones

across multiple parts of the app

between Friday night and Monday morning, I pushed 290 PRs

I had a weekend where I wanted to push some things forward materially

still a bit dangerous, but less dangerous

and I know the repo much better so I am less dangerous

two weekends ago, on the other repo, where all my tooling is more mature

yes this is exactly the way

I want to get you all in the right mood of what the "very near future looks like"

as I had mentioned a few weeks ago, I speak from painful experience, I have gone through this whole cycle on a different repo starting in January and have 100% certainty how it will play out

because then we are creating new future issues from ourselves

and even with staging in place, it is important that production does not fall 420 commits behind

so it is neither going to be 1 at a time nor 'all of staging'

but 8 was a big change in how we handle wallets, let's give it another look tomorrow, all of us with a fresh mind

a lot of them will be relatively straightforward and harmless and with tools to help review them, a human review will allow you to say: yup, deployments 1 to 7, 9, 12, 14,15,16 seem ok, lets get them to prod

what this looks like at "today's scale" not "tomorrow's scale" is that we might have 10-30 deployments per day to staging

i think we are going to have to cherry pick what goes from staging to prod

the exact same reasons apply here

could you @[simo] do a good job developing if you could never deploy any of your work to staging to see what you did? would it improve or worsen your coding?

the way to maximize their performance is to treat them like a human

in time though (and I have done it, so I am speaking from experience), they are good enough even to handle db migrations and test them (on staging)

we can start with the FE, where it less sensitive

and so we need tooling to avoid that

bc once we make this change, you will proceed to get lost in the deployments

this is going to need significantly more sophisticated workflows and tooling than you guys have now

human review comes in part 5 and is the gating factor for now to go to production

1/ bots E2E test their work on staging 2/ once bot says "we are good" 3/ we have some other specialized bots that also check repeating issues e.g. security, performance, etc (all this means that staging has to become a realistic environment with actual data) 4/ #3 reviewers may be send them back to #2 5/ once everyone is happy, then the bots keep a list that shows the various commits/deployments, rank them by blast radius, risk, what was touched, testing strategy

I actually think the end-state should be something like this tbh

that now you are going to have to manually discover

and fixed them

it would for sure have discovered new things

could have done an E2E review on staging by itself

every single PR I have pushed the last 24 hours

we let them deploy to staging

the pipeline is very straightforward

and they should do those before human review