Se7en Deployment Sins

A presentation at DevConf in September 2018 in Kraków, Poland by Philipp Krenn

Slide 1

Slide 1

Se7en Deployment Sins Philipp Krenn @xeraa

Slide 2

Slide 2

Developer

Slide 3

Slide 3

Slide 4

Slide 4

It depends...

Slide 5

Slide 5

1997: “Hello World!” 2017: “Cloud Native Distributed Containerized RESTful microservice based on ARMv7 w/ k8s Hello World! as a service.” — https://twitter.com/shanselman/status/923721516117778432

Slide 6

Slide 6

Slide 7

Slide 7

Gluttony is the overindulgence and overconsumption of anything to the point of waste.

Slide 8

Slide 8

Slide 9

Slide 9

Memory, Disk, Network

Slide 10

Slide 10

Windows 95 was 30Mb. Today we have web pages heavier than that! Windows 10 is 4Gb, which is 133 times as big. But is it 133 times as superior?

Slide 11

Slide 11

Google keyboard app routinely eats 150 Mb. Is an app that draws 30 keys on a screen really five times more complex than the whole Windows 95? — http://tonsky.me/blog/disenchantment/

Slide 12

Slide 12

"Windows 95 was 30 MB" is such an ignorant, obnoxious, trite take. a triple buffered framebuffer (which you want for smooth scrolling) for my 4K display is 70 MB in pixels alone. Obviously a complete webpage with precomposed textures would take more. — https://twitter.com/whitequark/status/1043747987342671872

Slide 13

Slide 13

640K ought to be enough for anybody

Slide 14

Slide 14

Slide 15

Slide 15

Slide 16

Slide 16

Slide 17

Slide 17

The JS Drinking Game: 1. Think of a noun. 2. Google "<noun>.js" 3. If a libary with that name exists, drink! #JavaScript #code — https://twitter.com/_ericelliott/status/ 746427779525279744

Slide 18

Slide 18

Slide 19

Slide 19

Slide 20

Slide 20

Slide 21

Slide 21

Microservices Nano, Pico, Femto, Atto, Yocto

Slide 22

Slide 22

Slide 23

Slide 23

Don’t forget - having legacy software is often a sign of success. Your business was successful to last long enough for your software to become legacy. — https://twitter.com/samnewman/status/ 929622273065136130

Slide 24

Slide 24

Microservices help if too many people too many dependencies not scalable

Slide 25

Slide 25

Will this solve all your problems?

Slide 26

Slide 26

Slide 27

Slide 27

Reminder: if you're building microservices, you're building a distributed system. That means, among other things,

Slide 28

Slide 28

each service has to be independently testable and deployable (including to your dev environment) and should degrade gracefully if it cannot reach its dependencies. — https://twitter.com/jezhumble/status/ 1021897540445196288

Slide 29

Slide 29

Sun Microsystems 8 fallacies of distributed computing

Slide 30

Slide 30

Slide 31

Slide 31

Slide 32

Slide 32

Jeff Hodges Notes on Distributed Systems for Young Bloods https://www.somethingsimilar.com/2013/01/14/noteson-distributed-systems-for-young-bloods/

Slide 33

Slide 33

The Five Nines Is it resilient? Nein. Is it available? Nein. Is it scalable? Nein. Is it reliable? Nein. Does it work? Nein. — https://twitter.com/marcvolovic/status/ 898022815668088832

Slide 34

Slide 34

At this point it would be easier if you told us when it DID work. — https://twitter.com/honest_update/status/ 930128629677416450

Slide 35

Slide 35

SOA done right?

Slide 36

Slide 36

Slide 37

Slide 37

Slide 38

Slide 38

SOAP: Simple Object Access Protocol. Except it's not Simple, it's Complicated. And it's not about Objects, it's about RPC. So really: CRAP — https://twitter.com/joewalnes/status/563691487586549761

Slide 39

Slide 39

Slide 40

Slide 40

We replaced our monolith with micro services so that every outage could be more like a murder mystery. — https://twitter.com/honest_update/status/ 651897353889259520

Slide 41

Slide 41

Slide 42

Slide 42

Here's a diagram of two microservices and their shared database. — https://twitter.com/mathiasverraes/status/ 711168935798902785

Slide 43

Slide 43

OH: "replace your shared monolithic DB with a shared monolithic stream store" — https://twitter.com/ntcoding/status/928902644764200960

Slide 44

Slide 44

In every distributed system architecture diagram, change the box labeled “Kafka” to “ESB” and immediately “The hot new thing” becomes “The legacy we need to get rid of”. Think about that. #decentralize — https://twitter.com/olivergierke/status/1044592822500904960

Slide 45

Slide 45

...don’t even consider microservices unless you have a system that’s too complex to manage as a monolith. The majority of software systems should be built as a single monolithic application.

Slide 46

Slide 46

Do pay attention to good modularity within that monolith, but don’t try to separate it into separate services. — http://martinfowler.com/bliki/MicroservicePremium.html

Slide 47

Slide 47

Slide 48

Slide 48

https://twitter.com/simonbrown/status/847339104874381312

Slide 49

Slide 49

The #microservices bandwagon exists because the name is cool. Monoliths just need marketing help… #MegaPlatform #UberContainer #StereoLith — https://twitter.com/sjmaple/status/724541627407147008

Slide 50

Slide 50

#Serverfull

Slide 51

Slide 51

A word of the week: Microlith - a huge enterprise app ported to one, fat Docker container. #MicroserviceJokes — https://twitter.com/rmehmandarov/status/ 865542627004788737

Slide 52

Slide 52

If you want to implement microservices, the 1st thing you need to do is implement aggregated logging. The 2nd is implement correlation IDs — https://twitter.com/samnewman/status/ 862255875125366786

Slide 53

Slide 53

Microservices can help large number of people go faster by allowing more autonomy. I remain to be convinced it does the same for small teams — https://twitter.com/samnewman/status/ 778118760226062336

Slide 54

Slide 54

Repeat after me I'm not Facebook, Google, or Amazon

Slide 55

Slide 55

Slide 56

Slide 56

Greed is applied to a very excessive or rapacious desire and pursuit of material possessions.

Slide 57

Slide 57

Slide 58

Slide 58

Slide 59

Slide 59

Sloth is sometimes defined as physical laziness, but spiritual laziness is emphasized.

Slide 60

Slide 60

Slide 61

Slide 61

Continuous Integration CI Build and test every push

Slide 62

Slide 62

Continuous Delivery CD Manual production deployment

Slide 63

Slide 63

Continuous Deployment CD Automated production deployment

Slide 64

Slide 64

Slide 65

Slide 65

Facebook switched to Continuous Delivery / Deployment in 04/2017 https://code.facebook.com/posts/270314900139291/ rapid-release-at-massive-scale/

Slide 66

Slide 66

Deploy multiple times per day if it fits your business model. Don’t deploy for the sake of deploying.

Slide 67

Slide 67

https://twitter.com/allspaw/status/925490726468771840

Slide 68

Slide 68

Slide 69

Slide 69

Slide 70

Slide 70

Slide 71

Slide 71

DDD (Russian: Davai Davai Deploy) Just deploy this sh*t already. Come on! — https://twitter.com/Steve_Jules/status/ 862247438551265280

Slide 72

Slide 72

Also Deadline Driven Deployment

Slide 73

Slide 73

Slide 74

Slide 74

Slide 75

Slide 75

https://twitter.com/ericqweinstein/status/529319496029589504

Slide 76

Slide 76

Slide 77

Slide 77

Volkswagen detects when your tests are being run in a CI server, and makes them pass. https://github.com/auchenberg/volkswagen https://github.com/hmlb/phpunit-vw

Slide 78

Slide 78

Slide 79

Slide 79

Separate code and config

Slide 80

Slide 80

Do not commit secrets HashiCorp Vault Ansible Vault ...

Slide 81

Slide 81

Slide 82

Slide 82

Lust is an intense and uncontrolled desire.

Slide 83

Slide 83

Always use the hottest

Slide 84

Slide 84

High-latency low-throughput append-only database with very expensive transaction commit protocols just doesn’t have the same ring to it as “Blockchain” does it? — https://twitter.com/jstogdill/status/1004448569615626240

Slide 85

Slide 85

#Database: garbage-in garbage-out #Blockchain: garbage-in garbagecarved-in-stone — https://twitter.com/nivertech/status/1033037299553103872

Slide 86

Slide 86

Slide 87

Slide 87

Slide 88

Slide 88

Docker: each docker container should just run one process Me: but your containers are 700+MB each Docker: you see why our mascot is a whale — https://twitter.com/classam/status/743333692462895104

Slide 89

Slide 89

Slide 90

Slide 90

https://turnoff.us/geek/ before-devops-afterdevops/

Slide 91

Slide 91

Slide 92

Slide 92

Slide 93

Slide 93

Slide 94

Slide 94

Slide 95

Slide 95

"A container is just a process"

Slide 96

Slide 96

Slide 97

Slide 97

containers will not fix your broken architecture you are welcome — https://twitter.com/littleidea/status/659445920954642432

Slide 98

Slide 98

Slide 99

Slide 99

You gotta love the docker-compose command line: docker-compose down -v Removes all volumes docker-compose -v down Prints the version of docker-compose — https://twitter.com/fhopf/status/971910920522760192

Slide 100

Slide 100

New dependencies of your artifacts JVM and base image

Slide 101

Slide 101

so.. using GlibC? How’s re-imaging all of your @Docker images going? — https://twitter.com/starbuxman/status/ 700591322177019904

Slide 102

Slide 102

Slide 103

Slide 103

Containerizing legacy applications

Slide 104

Slide 104

Microservices + containers Cargo cult

Slide 105

Slide 105

Slide 106

Slide 106

cargo culting comes from received but not integrated knowledge. because one doesn't grasp how a thing works, one simply goes through the motion of how one was told to do it. — https://twitter.com/GeePawHill/status/ 931941663886454784

Slide 107

Slide 107

If you are only picking Kubernetes because that's the way Google does it, then you should also consider writing your own filesystems. — https://twitter.com/kelseyhightower/status/ 741310392756887552

Slide 108

Slide 108

At this point you're crazy to think you can manage k8s yourself. I'm sure you've got engineers who want to. It's shiny and fun now, but the novelty wears off quick. And then you are left with an old snowflake — https://twitter.com/ibuildthecloud/status/ 1020371700934881280

Slide 109

Slide 109

Kubris - the belief that Kubernetes can do everything by itself — https://twitter.com/monadic/status/1007293757501378560

Slide 110

Slide 110

OH: “their infra is 50 shades of broken and they’re now setting up their own frankenetes cluster hoping it’ll fix everything” — https://twitter.com/copyconstruct/status/ 935317466573504512

Slide 111

Slide 111

We've been so beaten into thinking we need to care about the speed-ofdelivery with containers, we lost focus somewhere on prod reliability — https://twitter.com/danveloper/status/858372256174669824

Slide 112

Slide 112

https:// www.monkeyuser.com/ 2018/tech-debt/

Slide 113

Slide 113

What's wrong here? For a stateful production service $ docker run -p 9200:9200 -p 9300:9300 -e "http.host=0.0.0.0" -e "transport.host=0.0.0.0" -e "discovery.zen.ping.unicast.hosts=..." docker.elastic.co/elasticsearch/elasticsearch

Slide 114

Slide 114

Zombies ideas that should have been killed by evidence, but keep shambling along

Slide 115

Slide 115

Run Elasticsearch as root

Slide 116

Slide 116

Cockroaches claims that disappear for a while when proved wrong, but just keep on coming back

Slide 117

Slide 117

Make JAR not WAR

Slide 118

Slide 118

YAML

Slide 119

Slide 119

https://twitter.com/Caged/status/1039937162769096704

Slide 120

Slide 120

One more thing Serverless

Slide 121

Slide 121

Serverless

Slide 122

Slide 122

Slide 123

Slide 123

Yesterday I have bird and bee discussion with junior devops for make realize cloud instance is born of real hardware. — https://twitter.com/DEVOPS_BORAT/status/ 289777231683788801

Slide 124

Slide 124

Slide 125

Slide 125

I felt a great disturbance in the cloud, as if millions of microservices suddenly cried out in terror, and were suddenly serverless. — https://twitter.com/kennybastani/status/ 858021145890762755

Slide 126

Slide 126

2016: Serverless Architecture 2017: Codeless Architecture 2018: Architectureless Architecture — https://twitter.com/benbjohnson/status/ 746049032699600897

Slide 127

Slide 127

by 2025, 30% of net new technical debt will be serverless — https://twitter.com/mstine/status/972207185005633536

Slide 128

Slide 128

PaaS reborn?

Slide 129

Slide 129

If your PaaS can efficiently start instances in 20ms that run for half a second, then call it serverless. — https://twitter.com/adrianco/status/736553530689998848

Slide 130

Slide 130

Every vendor that can't compete in Cloud chooses "Hybrid" as their strategy - Opinion's law of Cloud. — https://twitter.com/cloud_opinion/status/ 594215778947362816

Slide 131

Slide 131

Slide 132

Slide 132

Pride is identified as believing that one is essentially better than others.

Slide 133

Slide 133

Slide 134

Slide 134

"I know, we need DevDevOpsOps!" — https://twitter.com/ agile_memes/status/ 1019694518050754560

Slide 135

Slide 135

Slide 136

Slide 136

DevOps is not a tool | certificate | job title | team

Slide 137

Slide 137

Slide 138

Slide 138

Slide 139

Slide 139

To make error is human. To propagate error to all server in automatic way is #devops. — https://twitter.com/DEVOPS_BORAT/status/ 41587168870797312

Slide 140

Slide 140

DevOps is a software engineering culture and practice of putting horrors into containers and then talking about Kubernetes at conferences. — https://twitter.com/sadserver/status/1011652439303262208

Slide 141

Slide 141

Slide 142

Slide 142

Envy is the discontent towards someone's traits, status, abilities, or rewards.

Slide 143

Slide 143

Craft everything yourself!

Slide 144

Slide 144

! Infrastructure vs

Slide 145

Slide 145

All my servers are cattle ... except for the redis instance. That one gets health care and Christmas gifts. — https://twitter.com/ajordens/status/801847570259509248

Slide 146

Slide 146

Slide 147

Slide 147

Slide 148

Slide 148

"Every company has a test environment. Some are lucky enough to also have a production environment." — https://twitter.com/lowrykoz/status/1017850159198294022

Slide 149

Slide 149

Avoid Drift

Slide 150

Slide 150

Slide 151

Slide 151

I like the "ssh tags the instance for garbage collection" model as a step; it lets you migrate a team to new habits. — https://twitter.com/ferlatte/status/905984786082050048

Slide 152

Slide 152

Slide 153

Slide 153

Wrath also known as "rage", may be described as inordinate and uncontrolled feelings of hatred and anger.

Slide 154

Slide 154

How I Hacked My Morning Routine: I wiped a production database and now I am 100% definitely awake — https://twitter.com/iamdevloper/status/ 1034824761833136128

Slide 155

Slide 155

Slide 156

Slide 156

How do you monitor your system? End users.

Slide 157

Slide 157

UDP : User Distress Protocol. — https://twitter.com/thisisvibs/status/1022323915417210880

Slide 158

Slide 158

pretty awesome when monitoring tools say zero overhead. Might as well say negative overhead.. I mean if you are going to lie at least add value! — https://twitter.com/adrianfcole/status/ 1024140189331906560

Slide 159

Slide 159

Slide 160

Slide 160

What we really do when a server is down. #DevOps — https://twitter.com/ mattiasgeniar/status/ 1039892272643489792

Slide 161

Slide 161

Conclusion

Slide 162

Slide 162

What did we learn today? It depends...

Slide 163

Slide 163

A good developer is like a werewolf: Afraid of silver bullets. — https://twitter.com/codepitbull/status/ 784691906005635072

Slide 164

Slide 164

Think back five years ago. How many technical opinions did you deeply hold which you now repudiate? Now think about the present. How many opinions do you deeply hold today?

Slide 165

Slide 165

We are always in a state of progressive wrongness. Don't cling too tightly to what you believe today. — https://twitter.com/djspiewak/status/ 1040247203648819200

Slide 166

Slide 166

Slide 167

Slide 167

Questions? Confessions? @xeraa