فهرست مطالب

• Experimentation for Engineers
• Preface
• Acknowledgments
• About this book
• About the author
• About the cover illustration
• 1: Optimizing systems by experiment
• 1.1 Examples of engineering workflows
• 1.2 Measuring by experiment
• 1.3 Why are experiments necessary?
• Summary
• 2: A/B testing: Evaluating a modification to your system
• 2.1 Take an ad hoc measurement
• 2.2 Take a precise measurement
• 2.3 Run an A/B test
• Summary
• 3: Multi-armed bandits: Maximizing business metrics while experimenting
• 3.1 Epsilon-greedy: Account for the impact of evaluation on business metrics
• 3.2 Evaluating multiple system changes simultaneously
• 3.3 Thompson sampling: A more efficient MAB algorithm
• Summary
• 4: Response surface methodology: Optimizing continuous parameters
• 4.1 Optimize a single continuous parameter
• 4.2 Optimizing two or more continuous parameters
• Summary
• 5: Contextual bandits: Making targeted decisions
• 5.1 Model a business metric offline to make decisions online
• 5.2 Explore actions with epsilon-greedy
• 5.3 Explore parameters with Thompson sampling
• 5.4 Validate the contextual bandit
• Summary
• 6: Bayesian optimization: Automating experimental optimization
• 6.1 Optimizing a single compiler parameter, a visual explanation
• 6.2 Model the response surface with Gaussian process regression
• 6.3 Optimize over an acquisition function
• 6.4 Optimize all seven compiler parameters
• Summary
• 7: Managing business metrics
• 7.1 Focus on the business
• 7.2 Define business metrics
• 7.3 Trade off multiple business metrics
• Summary
• 8: Practical considerations
• 8.1 Violations of statistical assumptions
• 8.2 Don’t stop early
• 8.3 Control family-wise error
• 8.4 Be aware of common biases
• 8.5 Replicate to validate results
• 8.6 Wrapping up
• Summary
• Appendix A—Linear regression and the normal equations
• Appendix B—One factor at a time
• Appendix C—Gaussian process regression
• Index