Se7en Deadly Deployment Sins
A presentation at DevDays Vilnius in in Vilnius, Lithuania by Philipp Krenn
Se7en Deployment Sins Philipp Krenn @xeraa
Developer Advocate
Gluttony is the overindulgence and overconsumption of anything to the point of waste.
Memory PermGen Space / Metaspace
JAR / WAR Disk and network
Diet aws-java-sdk-1.8.12.jar 12.9 M aws-java-sdk-1.11.335.jar 3.4 K
people arguing against fat JARs (only ship the business logic), but perfectly ! ne with containers - shipping even the JVM ! #RigaDevDays — https://twitter.com/xeraa/status/864122131768119296
640K ought to be enough for anybody
Microservices Nano, Pico, Femto, Atto, Yocto
Don’t forget - having legacy software is often a sign of success. Your business was successful to last long enough for your software to become legacy. — https://twitter.com/samnewman/status/ 929622273065136130
Yes, it helps if too many people too many dependencies not scalable
Will this solve all your problems?
Sun Microsystems 8 fallacies of distributed computing
Jeff Hodges Notes on Distributed Systems for Young Bloods https://www.somethingsimilar.com/2013/01/14/notes- on-distributed-systems-for-young-bloods/
The Five Nines Is it resilient? Nein. Is it available? Nein. Is it scalable? Nein. Is it reliable? Nein. Does it work? Nein. — https://twitter.com/marcvolovic/status/ 898022815668088832
At this point it would be easier if you told us when it DID work. — https://twitter.com/honest_update/status/ 930128629677416450
SOA done right?
SOAP: Simple Object Access Protocol. Except it's not Simple, it's Complicated. And it's not about Objects, it's about RPC. So really: CRAP — https://twitter.com/joewalnes/status/563691487586549761
We replaced our monolith with micro services so that every outage could be more like a murder mystery. — https://twitter.com/honest_update/status/ 651897353889259520
Here's a diagram of two microservices and their shared database. — https://twitter.com/mathiasverraes/status/ 711168935798902785
OH: "replace your shared monolithic DB with a shared monolithic stream store" — https://twitter.com/ntcoding/status/928902644764200960
...don’t even consider microservices unless you have a system that’s too complex to manage as a monolith. The majority of software systems should be built as a single monolithic application.
Do pay attention to good modularity within that monolith, but don’t try to separate it into separate services. — http://martinfowler.com/bliki/MicroservicePremium.html
https://twitter.com/simonbrown/status/847339104874381312
The #microservices bandwagon exists because the name is cool. Monoliths just need marketing help… #MegaPlatform #UberContainer #StereoLith — https://twitter.com/sjmaple/status/724541627407147008
#Serverfull
A word of the week: Microlith - a huge enterprise app ported to one, fat Docker container. #MicroserviceJokes — https://twitter.com/rmehmandarov/status/ 865542627004788737
If you want to implement microservices, the 1st thing you need to do is implement aggregated logging. The 2nd is implement correlation IDs — https://twitter.com/samnewman/status/ 862255875125366786
Microservices can help large number of people go faster by allowing more autonomy. I remain to be convinced it does the same for small teams — https://twitter.com/samnewman/status/ 778118760226062336
Repeat after me I'm not Facebook, Google, or Amazon
Greed is applied to a very excessive or rapacious desire and pursuit of material possessions.
Sloth is sometimes defined as physical laziness, but spiritual laziness is emphasized.
Continuous life or death cycle?
Continuous Integration CI Build and test every push
Continuous Delivery CD Manual production deployment
Continuous Deployment CD Automated production deployment
https://twitter.com/allspaw/status/925490726468771840
Facebook switched to Continuous Delivery / Deployment in 04/2017 https://code.facebook.com/posts/270314900139291/ rapid-release-at-massive-scale/
DDD (Russian: Davai Davai Deploy) - Just deploy this sh*t already. Come on! — https://twitter.com/Steve_Jules/status/ 862247438551265280
https://twitter.com/ericqweinstein/status/529319496029589504
Separate code and config
Do not commit secrets HashiCorp Vault Ansible Vault ...
Lust is an intense and uncontrolled desire.
Always use the hottest !
Docker: each docker container should just run one process Me: but your containers are 700+MB each Docker: you see why our mascot is a whale — https://twitter.com/classam/status/743333692462895104
https://turnoff.us/geek/before-devops- after-devops/
"A container is just a process"
containers will not ! x your broken architecture you are welcome — https://twitter.com/littleidea/status/659445920954642432
so.. using GlibC? How’s re-imaging all of your @Docker images going? — https://twitter.com/starbuxman/status/ 700591322177019904
You gotta love the docker-compose command line: docker-compose down -v Removes all volumes docker-compose -v down Prints the version of docker-compose — https://twitter.com/fhopf/status/971910920522760192
New dependencies of your artifacts JVM and base image
Containerizing legacy applications
Microservices + containers Cargo cult
cargo culting comes from received but not integrated knowledge.because one doesn't grasp how a thing works, one simply goes through the motion of how one was told to do it. — https://twitter.com/GeePawHill/status/ 931941663886454784
If you are only picking Kubernetes because that's the way Google does it, then you should also consider writing your own ! lesystems. — https://twitter.com/kelseyhightower/status/ 741310392756887552
Siri, mute every person who told me to use docker instead of helping me solve my actual problem, as if I didn't understand my question. — https://twitter.com/aphyr/status/730571211864858624
We've been so beaten into thinking we need to care about the speed-of- delivery with containers, we lost focus somewhere on prod reliability — https://twitter.com/danveloper/status/858372256174669824
What's wrong here? For a stateful production service $ docker run -p 9200:9200 -p 9300:9300 -e "http.host=0.0.0.0" -e "transport.host=0.0.0.0" -e "discovery.zen.ping.unicast.hosts=..." docker.elastic.co/elasticsearch/elasticsearch
Zombies ideas that should have been killed by evidence, but keep shambling along
Run Elasticsearch as root
Cockroaches claims that disappear for a while when proved wrong, but just keep on coming back
Make JAR not WAR
One more thing Serverless
Serverless
Yesterday I have bird and bee discussion with junior devops for make realize cloud instance is born of real hardware. — https://twitter.com/DEVOPS_BORAT/status/ 289777231683788801
I felt a great disturbance in the cloud, as if millions of microservices suddenly cried out in terror, and were suddenly serverless. — https://twitter.com/kennybastani/status/ 858021145890762755
2016: Serverless Architecture 2017: Codeless Architecture 2018: Architectureless Architecture — https://twitter.com/benbjohnson/status/ 746049032699600897
by 2025, 30% of net new technical debt will be serverless — https://twitter.com/mstine/status/972207185005633536
PaaS reborn?
If your PaaS can e ! ciently start instances in 20ms that run for half a second, then call it serverless. — https://twitter.com/adrianco/status/736553530689998848
Pride is identified as believing that one is essentially better than others.
DevOps is not a technology
To make error is human. To propagate error to all server in automatic way is #devops. — https://twitter.com/DEVOPS_BORAT/status/ 41587168870797312
Envy is the discontent towards someone's traits, status, abilities, or rewards.
Craft everything yourself!
Example: Java https://start.spring.io
"Nobody uses Maven. Maven uses you" @venkat_s at #Devoxx — https://twitter.com/mariofusco/status/ 927899858056425472
Infrastructure Pets vs Cattle
All my servers are cattle ... except for the redis instance. That one gets health care and Christmas gifts. — https://twitter.com/ajordens/status/801847570259509248
Ansible Puppet Chef Terraform
Avoid Drift
I like the "ssh tags the instance for garbage collection" model as a step; it lets you migrate a team to new habits. — https://twitter.com/ferlatte/status/905984786082050048
Wrath also known as "rage", may be described as inordinate and uncontrolled feelings of hatred and anger.
Log & Monitor
Conclusion
A good developer is like a werewolf: Afraid of silver bullets. — https://twitter.com/codepitbull/status/ 784691906005635072
OH: “their infra is 50 shades of broken and they’re now setting up their own frankenetes cluster hoping it’ll ! x everything” ! " — https://twitter.com/copyconstruct/status/ 935317466573504512
https://youtu.be/zoGyuiyCUAQ
Questions? Confessions? @xeraa
Do you have something to confess? Or are you still in denial — knowing that you're doing something wrong, but you cannot say for sure what it is? In this talk, we'll go through our deployment sins and how to avoid them:
- Gluttony: Adding all the dependencies — both in size and number.
- Greed: Yes, I want to use the biggest and slowest application server possible.
- Sloth: Continuous Deployment or Delivery — who would need that?!
- Lust: Of course I'm using containers, microservices, and every latest trend for every project!
- Pride: Once something is in production it's purely an ops problem.
- Envy: Why should I rely on an external library, when I can lovingly handcraft it all myself?
- Wrath: Logging and monitoring are for the faint of heart!
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
Notes on Distributed Systems for Young Bloods
I’ve been thinking about the lessons distributed systems engineers learn on the job. A great deal of our instruction is through scars made by mistakes made in production traffic. These scars are useful reminders, sure, but it’d be better to have more engineers with the full count of their fingers.
New systems engineers will find the Fallacies of Distributed Computing and the CAP theorem as part of their self-education. But these are abstract pieces without the direct, actionable advice the inexperienced engineer needs to start moving. It’s surprising how little context new engineers are given when they start out.
Below is a list of some lessons I’ve learned as a distributed systems engineer that are worth being told to a new engineer. Some are subtle, and some are surprising, but none are controversial. This list is for the new distributed systems engineer to guide their thinking about the field they are taking on. It’s not comprehensive, but it’s a good beginning.
Buzz and feedback
Here’s what was said about this presentation on social media.
