Presentation by Damon Edwards at DevOpsDays Rome 2012.
Topics:
• What do we mean by "Culture"?
• DevOps Vision defined
1. See the system
2. Focus on flow
3. Recognize feedback loops
4. Look for continuous improvement opportunities
• Examples of techniques that high performing companies use
7. Where do we put our efforts?
ORGANIZATION
ORGANIZATION ORGANIZATION ORGANIZATION
TEAM TEAM
TEAM TEAM TEAM
TEAM TEAM TEAM
TEAM
TEAM TEAM TEAM
TEAM
TEAM TEAM TEAM
TEAM
TEAM TEAM TEAM
EASIER MORE DIFFICULT VERY DIFFICULT
8. Where do we put our efforts?
ORGANIZATION ORGANIZATION ORGANIZATION
TEAM TEAM TEAM TEAM TEAM TEAM
TEAM TEAM TEAM
TEAM TEAM TEAM
TEAM TEAM TEAM
EASIER MORE DIFFICULT VERY DIFFICULT
9. Where do we put our efforts?
ORGANIZATION ORGANIZATION ORGANIZATION
TEAM TEAM TEAM TEAM TEAM TEAM
TEAM TEAM TEAM
TEAM TEAM TEAM
TEAM TEAM TEAM
EASIER MORE DIFFICULT VERY DIFFICULT
10. Where do we put our efforts?
ORGANIZATION ORGANIZATION ORGANIZATION
TEAM TEAM TEAM TEAM TEAM TEAM
TEAM TEAM TEAM
TEAM TEAM TEAM
TEAM TEAM TEAM
EASIER MORE DIFFICULT VERY DIFFICULT
11. Where do we put our efforts?
ORGANIZATION ORGANIZATION ORGANIZATION
TEAM TEAM TEAM TEAM TEAM TEAM
TEAM TEAM TEAM
TEAM TEAM TEAM
TEAM TEAM TEAM
EASIER MORE DIFFICULT VERY DIFFICULT
DevOps problems live here!
12. State of the Art DevOps Culture Talk in One Slide!
13. State of the Art DevOps Culture Talk in One Slide!
14. State of the Art DevOps Culture Talk in One Slide!
+
15. What your mother told you was culture
Culture (n.):
Culture (adj.):
16. “Culture” really means “Organizational Culture”
“Set of shared mental assumptions
that guide interpretation and action in
organizations by defining appropriate
behavior for various situations.”
17. “Culture” really means “Organizational Culture”
“Set of shared mental assumptions
that guide interpretation and action in
organizations by defining appropriate
behavior for various situations.”
Why we do it the way we do it.
20. Be not afraid!
• It’s not as squishy as it seems
• There is actually some solid academic
work to grab onto
• There are tools to at least describe, if not
comprehend, organizational culture
28. What to do?
• Option #1: Change your company
If you can’t then...
29. What to do?
• Option #1: Change your company
If you can’t then...
• Option #2: Change your company
30. What to do?
• Option #1: Change your company
If you can’t then...
• Option #2: Change your company
If you can’t then...
31. What to do?
• Option #1: Change your company
If you can’t then...
• Option #2: Change your company
If you can’t then...
• Option #3: (see Option #1)
37. What is DevOps Vision?
1. See the system
2. Focus on flow
38. What is DevOps Vision?
1. See the system
2. Focus on flow
3. Recognize feedback loops
39. What is DevOps Vision?
1. See the system
2. Focus on flow
3. Recognize feedback loops
4. Look for continuous improvement opportunities
40. What is DevOps Vision?
™
EA
THE BIG ID
1. See the system
2. Focus on flow
3. Recognize feedback loops
4. Look for continuous improvement opportunities
41. Infuse DevOps Vision into every cultural artifact
• values 1. See the system
• vision
• norms 2. Focus on flow
• language
• systems
3. Recognize feedback loops
• symbols
• beliefs
4. Look for continuous improvement
• habits opportunities
42. Now is the time to make the business case for
DevOps behavior changes!
1. See the system
• values
2. Focus on flow • vision Better TTM
• norms
• language Better Quality
• systems
3. Recognize feedback loops
• symbols
• beliefs
• habits
4. Look for continuous
improvement opportunities
45. Infusing DevOps Vision into an Organization
• What doesn’t work: The Big Bang™
• What does work: Recruit, Educate, Drill, Repeat
46. Now for some interesting examples on
bringing DevOps Vision to life...
47. Banish the word “done”
DONE
Why?: Owner: Dev / Project Mgmt
Drives “system thinking” and
Impact: Developers, QA
“focus on flow”
Difficulty: Easy
48. Banish the word “done”
DONE
• You run a service. It never stops, so you are never
“done”.
Why?: Owner: Dev / Project Mgmt
Drives “system thinking” and
Impact: Developers, QA
“focus on flow”
Difficulty: Easy
49. Banish the word “done”
DONE
• You run a service. It never stops, so you are never
“done”.
• “Code complete” is an anti-pattern, not a milestone
Why?: Owner: Dev / Project Mgmt
Drives “system thinking” and
Impact: Developers, QA
“focus on flow”
Difficulty: Easy
50. Banish the word “done”
DONE
• You run a service. It never stops, so you are never
“done”.
• “Code complete” is an anti-pattern, not a milestone
• At best, you have fulfilled your most recent obligation
Why?: Owner: Dev / Project Mgmt
Drives “system thinking” and
Impact: Developers, QA
“focus on flow”
Difficulty: Easy
51. Make operational requirements first class citizens
Why?: Owner: Program Management
Drives systems thinking, improves
Impact: Schedule Owner
predictability of flow, reduces
technical debt creation Difficulty: Easy
52. Make operational requirements first class citizens
• Operational requirements (“non-functional
requirements) are just as important to running
service as any business feature requirement
Why?: Owner: Program Management
Drives systems thinking, improves
Impact: Schedule Owner
predictability of flow, reduces
technical debt creation Difficulty: Easy
53. Make operational requirements first class citizens
• Operational requirements (“non-functional
requirements) are just as important to running
service as any business feature requirement
• Include those operational requirements in whatever
tools and processes your business feature
requirements go through
Why?: Owner: Program Management
Drives systems thinking, improves
Impact: Schedule Owner
predictability of flow, reduces
technical debt creation Difficulty: Easy
54. Visualize the flow
Why?: Owner: Release (?)
Drives systems thinking and
focusing on the flow, makes Impact: Everyone
bottlenecks obvious
Difficulty: Easy
55. Collaboratively map the value stream Core business service delivery process
Steering
Current state value stream map
Committee
Erica Switzer
Full L/T Lead time
Projects /
Expedites P/T Process time
H/C Head count
S/R Scrap rate
Business Analyst
Erica Switzer
D Defects
Estimates
and EP Extra processes
requirements M Motion
PD Partially done
Project Manager TS Task switching
W Waiting
Erica Switzer
Business
Change
Creative
request
Daily Scrum Change
Program Management
meeting Request QA Control Forms
Mx3 build Development
Dx3 Develop Technical
Specification Daily Scrum Deployment
web assets meeting QA
Meeting Business Systems
Jira issues RT tickets Support Release Engineering
L/T 6 weeks
Web Assets, Deployment
P/T 4 weeks Technical Operations
flows and Jira issues Document Stan Walters
H/C 12 wireframes
S/R 5%
Bob McNulty Wx2 PDx4 Wx2 RT tickets Change
TSx5 Dx2
Delta Development Development D Release Control
Development features and Build and deploy Integration testing Build and deploy
Set of fixes
EP service L/T 8 weeks L/T 2 days L/T 1 day L/T 8 hours
Cut project branch branches P/T 4 weeks P/T 4 hours P/T 1 day P/T 7 hours
H/C 5 H/C 1 H/C 5 H/C 1 W
D D Mx2
S/R 25% S/R 10% S/R 95% S/R 2% PDx3 D Mx8 Dx2
L/T 15 minutes
Joe Harbaugh Joe Harbaugh Joe Harbaugh Fernando Gomez EP PD Deploy to EP
P/T 5 minutes Acceptance testing Feature testing Tested Merge project
Deploy to Staging Staging testing
H/C 1 packages Production branch
S/R 0%
L/T 5 days L/T 4 weeks L/T 6 days L/T 1 week L/T 9 hours L/T 30 minutes
Chris Holmes
Development P/T 5 hours P/T 2.5 weeks P/T 90 minutes P/T 3.5 days P/T 8 hours P/T 30 minutes
Environments H/C 2 H/C 4 H/C 3 H/C 4 H/C 6 H/C 1
Documented S/R 10% S/R 20% S/R 66% S/R 66% S/R 66% S/R 33%
test plans Raj Lee Raj Lee Frances Middleton Frances Middleton Allen Cannata Fernando Gomez
and cases
PDx2
QA Staging Production
Build test plan Build test cases Environments Environment Environment
L/T 5 days L/T 2 weeks
P/T 4 days P/T 8 days
H/C 1 H/C 2
S/R 10% S/R 20%
Deploy to Performance
Raj Lee Raj Lee
Performance testing
L/T 6 days L/T 1 week
P/T 90 minutes P/T 3 days
H/C 3 H/C 1
S/R 66% S/R 66%
Frances Middleton Allen Cannata
Performance
Environment
Process Time
4 weeks 4 weeks 4 hours 1 day 7 hours 5 hours 2.5 weeks 90 minutes = 62 days
3.5 days 8 hours
2 weeks 4 weeks 1.5 days 1 hours 4.5 days 1.5 weeks 5.75 days 1.5 days 1 hour Lead Time =
113 days
Why?: Owner: Anyone
Aligns understanding of the full
system, highlights wastes Impact: Everyone
(problems)
Difficulty: Low /Medium
56. Turn information flows into artifact flows
M
Service pack
review
email Current state value stream map
Customer
L/T = 56d
P/T = 7d communication L/T Lead time
H/C = 6 P/T Process time
S/R = 100% L/T = 28d
Suresh Wu P/T = 7d H/C Head count
crit bugs H/C = 1
S/R = S/R Scrap rate
M(2) W Stephen / Xi Customer
derived Release Program D Defects
reqs. Management
Release EP Extra processes
M W
Schedule
L/T = M Motion
Estimates PRD P/T = Product Program
H/C = Planning PD Partially done
S/R = TS Task switching
Erica Smith L/T = 105d
P/T = 46d W Waiting
PD H/C = 15
S/R = 100% Lockdown Technical Support
M(2) EP John Robert control Product Management
Engineering Engineering
Planning Cloud Services
Process Design
PRD Specs PRD
L/T = 45d EP(2) TS(2)
P/T = 18d Documentum
M EP H/C = 23 M(3) W(2)
S/R = PD
Preliminary Bob Smith ERR Cloud Services
Development Release BRD M(3) W
Release PRD
ERR Server
L/T = 45d Memos L/T = 60d
Provisioning
P/T = 21d P/T = 16d
H/C = 140 D MOP, SOP H/C = 3
QA L/T = 24d
S/R = S/R = 3%
Environment Remedy P/T = 4d
Bob Smith QA Forum M(3) EP Reggie / Carlos
Ticket H/C = 3
Ticket QA forum Engineering S/R = 50% BRD
checklist Release Jen Garza
M W EP Patch
L/T = 60d Calendar
D P/T = 1d Change Control EP
H/C = 1 README
QA Forum S/R = >5% Rollout Server
M EP(3) L/T = 42d
Ticket Victoria Doe Schedule Acceptance
Tasks P/T =
QA Test
H/C = L/T = 14d
S/R = P/T = 1d
L/T = 105d MOP MOP Peter Lee H/C = 4.5
P/T = 11d ERR S/R = 15%
H/C = 42 PD(3)
Lynn A. etc
S/R = Selective
Sam Young Promotion
L/T = 90d
P/T = 15d Remedy
New Targets H/C = 5 Ticket
S/R =
Steve Young
BTS
README
M W(2) EP Deploy Release
Full
M(2) PD(3)
Development Shared
Commits Build Release L/T = 90d
Drive Prod Packages P/T = 8d
L/T = 75d Shared Drive Promotion
P/T = 43d L/T = 1d Test H/C = 8
H/C = 130 P/T = 0.3d L/T = 60d S/R = 2%
S/R = H/C = 2 P/T = 0.2d XML Lewis S./Peter Y.
Production
Bob Smith S/R = 33% H/C = 1 M W(2) EP
John Doe S/R = >5% Single
Victoria Doe Image PD D(3) TS
Server
Why?: Owner: Development
Removes barriers to flow, smooths
Impact: Dev and Ops
handoffs, improves quality
Difficulty: High
57. Turn information flows into artifact flows
M
Service pack
review
email Current state value stream map
Customer
L/T = 56d
P/T = 7d communication L/T Lead time
H/C = 6 P/T Process time
S/R = 100% L/T = 28d
Suresh Wu P/T = 7d H/C Head count
crit bugs H/C = 1
S/R = S/R Scrap rate
M(2) W Stephen / Xi Customer
derived Release Program D Defects
reqs. Management
Release EP Extra processes
M W
Schedule
L/T = M Motion
Estimates PRD P/T = Product Program
H/C = Planning PD Partially done
S/R = TS Task switching
Erica Smith L/T = 105d
P/T = 46d W Waiting
PD H/C = 15
S/R = 100% Lockdown Technical Support
M(2) EP John Robert control Product Management
Engineering Engineering
Planning Cloud Services
Process Design
PRD Specs PRD
L/T = 45d EP(2) TS(2)
P/T = 18d Documentum
M EP H/C = 23 M(3) W(2)
S/R = PD
Preliminary Bob Smith ERR Cloud Services
Development Release BRD M(3) W
Release PRD
ERR Server
L/T = 45d Memos L/T = 60d
Provisioning
P/T = 21d P/T = 16d
H/C = 140 D MOP, SOP H/C = 3
QA L/T = 24d
S/R = S/R = 3%
Environment Remedy P/T = 4d
Bob Smith QA Forum M(3) EP Reggie / Carlos
Ticket H/C = 3
Ticket QA forum Engineering S/R = 50% BRD
checklist Release Jen Garza
M W EP Patch
L/T = 60d Calendar
D P/T = 1d Change Control EP
H/C = 1 README
QA Forum S/R = >5% Rollout Server
M EP(3) L/T = 42d
Ticket Victoria Doe Schedule Acceptance
Tasks P/T =
QA Test
H/C = L/T = 14d
S/R = P/T = 1d
L/T = 105d MOP MOP Peter Lee H/C = 4.5
P/T = 11d ERR S/R = 15%
H/C = 42 PD(3)
Lynn A. etc
S/R = Selective
Sam Young Promotion
L/T = 90d
P/T = 15d Remedy
New Targets H/C = 5 Ticket
S/R =
Steve Young
BTS
README
M W(2) EP Deploy Release
Full
M(2) PD(3)
Development Shared
Commits Build Release L/T = 90d
Drive Prod Packages P/T = 8d
L/T = 75d Shared Drive Promotion
P/T = 43d L/T = 1d Test H/C = 8
H/C = 130 P/T = 0.3d L/T = 60d S/R = 2%
S/R = H/C = 2 P/T = 0.2d XML Lewis S./Peter Y.
Production
Bob Smith S/R = 33% H/C = 1 M W(2) EP
John Doe S/R = >5% Single
Victoria Doe Image PD D(3) TS
Server
Why?: Owner: Development
Removes barriers to flow, smooths
Impact: Dev and Ops
handoffs, improves quality
Difficulty: High
58. Turn information flows into artifact flows
M
Service pack
review
email Current state value stream map
Customer
L/T = 56d
P/T = 7d communication L/T Lead time
H/C = 6 P/T Process time
S/R = 100% L/T = 28d
Suresh Wu P/T = 7d H/C Head count
crit bugs H/C = 1
S/R = S/R Scrap rate
M(2) W Stephen / Xi Customer
derived Release Program D Defects
reqs. Management
Release EP Extra processes
M W
Schedule
L/T = M Motion
Estimates PRD P/T = Product Program
H/C = Planning PD Partially done
S/R = TS Task switching
Erica Smith L/T = 105d
P/T = 46d W Waiting
PD H/C = 15
S/R = 100% Lockdown Technical Support
M(2) EP John Robert control Product Management
Engineering Engineering
Planning Cloud Services
Process Design
PRD Specs PRD
L/T = 45d EP(2) TS(2)
P/T = 18d Documentum
M EP H/C = 23 M(3) W(2)
S/R = PD
Preliminary Bob Smith ERR Cloud Services
Development Release BRD M(3) W
Release PRD
ERR Server
L/T = 45d Memos L/T = 60d
Provisioning
P/T = 21d P/T = 16d
H/C = 140 D MOP, SOP H/C = 3
QA L/T = 24d
S/R = S/R = 3%
Environment Remedy P/T = 4d
Bob Smith QA Forum M(3) EP Reggie / Carlos
Ticket H/C = 3
Ticket QA forum Engineering S/R = 50% BRD
checklist Release Jen Garza
M W EP Patch
L/T = 60d Calendar
D P/T = 1d Change Control EP
H/C = 1 README
QA Forum S/R = >5% Rollout Server
M EP(3) L/T = 42d
Ticket Victoria Doe Schedule Acceptance
Tasks P/T =
QA Test
H/C = L/T = 14d
S/R = P/T = 1d
L/T = 105d MOP MOP Peter Lee H/C = 4.5
P/T = 11d ERR S/R = 15%
H/C = 42 PD(3)
Lynn A. etc
S/R = Selective
Sam Young Promotion
L/T = 90d
P/T = 15d Remedy
New Targets H/C = 5 Ticket
S/R =
Steve Young
BTS
README
M W(2) EP Deploy Release
Full
M(2) PD(3)
Development Shared
Commits Build Release L/T = 90d
Drive Prod Packages P/T = 8d
L/T = 75d Shared Drive Promotion
P/T = 43d L/T = 1d Test H/C = 8
H/C = 130 P/T = 0.3d L/T = 60d S/R = 2%
S/R = H/C = 2 P/T = 0.2d XML Lewis S./Peter Y.
Production
Bob Smith S/R = 33% H/C = 1 M W(2) EP
John Doe S/R = >5% Single
Victoria Doe Image PD D(3) TS
Server
Why?: Owner: Development
Removes barriers to flow, smooths
Impact: Dev and Ops
handoffs, improves quality
Difficulty: High
59. Metrics with a mission: Situational Awareness
Organization
& Process
Data
Application Situational Infrastructure
Data Awareness Data
Business
Data
Why?: Owner: Metrics (?)
Feedback. Aligns organizational
Impact: Everyone
efforts.
Difficulty: Medium
60. Turn everything into a change correlation chart
Why?: Owner: Metrics (?)
Raise awareness of change. Impact: Everyone
Difficulty: Medium
61. Paint the walls with awareness
Why?: Owner: Metrics (?)
Feedback. Aligns organizational
Impact: Everyone
efforts.
Difficulty: Easy
62. Realign Ownership and Control
Why?: Owner: Technology Managers
Removes bottlenecks (improves
flow), pushes the pain forward, aligns Impact: Everyone
activities with those best suited to
execute Difficulty: Medium / High
63. Realign Ownership and Control
• Dev owns... uptime for their own code
Ops owns... uptime for platforms and tooling
Why?: Owner: Technology Managers
Removes bottlenecks (improves
flow), pushes the pain forward, aligns Impact: Everyone
activities with those best suited to
execute Difficulty: Medium / High